This is a pretty intense project, it’s great to see so many community members helping out!
From a feasibility standpoint, it seems like the desired outcome is well within the capabilities of the RPi platform. I understand your concerns given the project’s scale, and there are a few things I think are worth considering before you start.
You mentioned the containers acting as semi-Faraday Cages, this could be highly beneficial if the labs are physically close given that the intra-lab communication is done via ethernet, it may well reduce interference from other labs.
You mentioned you want the PicoWs to have individual static IP addresses. I fear this type of solution will be “clunky”, for lack of a better term. I imagine it will also be quite annoying to expand the system were you to increase the size in future. That being said, I understand your reasoning given you want each PicoW to both send and receive packets. It would be worth looking into other solutions, such as software network multiplexers.
I have seen both TCP and UDP mentioned within the thread. Given the desired high data transfer rate I am of the opinion that UDP is the better choice here (which I believe was the protocol @Clem73188 mentioned in the original post), especially if you were to use a solution like a multiplexer mentioned above, which may increase the size of the data packets (SoF/EoF destination info, etc.).
Keep in mind these are just concerns that came to mind while reading the thread. At the end of the day complex networking like this often comes with many minor complications. I’ll keep this in mind and update the thread if I come up with any solutions in the coming days.
Hoping this helped, and good luck with the project!
@Clem73188 I had a couple of ideas, so I gave this a go again.
First idea did not work out, no need to say anymore.
Second idea did and no need for complex routing tables. So this is what I did.
Installed RPi OS Bookworm 64 Lite using the Raspberry Pi Imager program.
Started the Pi and installed RaspAP as per their official web site.
Connected to RaspAP application on the Pi via its web page with a PC.
Enabled Bridged AP Mode for the WiFi HotSpot and restarted the HotSpot.
At this point you lose connectivity to the RaspAP application web page.
Following the instructions on the official RaspAP web site was able to connect to the RaspAP application again to check it was working ok.
Now setup static IP addresses for the PicoW’s on my internet modem router.
Started the programs on the PicoW’s to connect to the RaspAP application HotSpot on the Pi.
Worked nicely.
Was able to ping the PicoW’s from the PC.
Proving the Pi was acting as a bridge between the WiFi and the PC network.
What is happening to make it work.
Essentially the internet modem router is managing all connections and static IP addresses. It is handling all routing tasks. So all devices can talk to each other.
How to separate the networks.
First; the private IP address range 192.168.0.0 to 192.168.255.255 is 256 contiguous class C networks according to the WiKi page for IP addresses.
Normally for internet modem routers the address range is 192,168.0.xxx or 192.168.1.xxx giving home users close to 256 possible devices, but some in that range are reserved.
With your set up you would allocate one of the 256 contiguous class C numbers to each container. ie 192.168.100.xxx or 192.168.50.xxx or 192.168.200.xxx and so on.
Each PicoW in the container would have the same number, allowing up to 256 devices in each container.
Internet modem routers use a subnet mask of 255.255.255.0. Allowing only devices with the same 3 numbers to communicate. But by using 255.255.0.0 you should be able to access all 65,536 possible devices.
I need to test this part, starting and stopping the internet modem router causes family problems so I need a quiet time to do it, if you know what I mean.
The test I did was with the following IP’s.
Internet modem router = 192.168.1.1
PC = 192.168.1.107 dynamic
Pi 4 = 192.168.1.103 dynamic (could be static, just have to put it in the router table)
PicoW One = 192.168.1.210 static
PicoW Two = 192.168.1.212 static
PicoW’s transfer UDP packets ok, PC can ping both as you would expect.
Next test will be to assign a different network to each with the Pi on the same one.
Hey… at last something I know a bit about
Just want to comment on Jim’s post, maybe Clem already knows this stuff…
Jim’s idea of allocating a different Class C network to each container seems reasonable…, but I think Jim goes off the rails a little with setting the netmask to 255.255.0.0, I wanted to ensure you are all aware of what is actually going on to avoid any confusion.
The netmask is a means or partitioning an IP address into two distinct parts: the network address, and the host address. Most common is the 255.255.255.0 mask, which means the first 3 octets identify the network, and the last octet identifies each of the possible 255 hosts that can connect to that network. FIne… this is used very widely be default.
When you switch to 255.255.0.0, what you get is ONE network address… ONE subnet, if you like, with up to 65535 different hosts (in this case, Picos), which ALL LIVE ON THE SAME NETWORK.
To split things up as JIm had in mind, with a different Class C network for each shipping container, you would really need a router with a whole stack of different interfaces, each one provisioning a different CLass C subnet. The router would then do the interconnections between subnets as required. No routing required between hosts inside a single container/subnet… but between containers… yep.
So Jim is right in saying 255.255.0.0 might be what you need… just know that the concept of having a Class C address allocated to each container kind of diverges from the implementation of a single Class B network. Don’t confuse the two!
You mention you need something fast and reliable, will this be used in control of these systems or simple logging?
If any control is involved I would switch to a wired system. Using any Area-Network protocol like MODBUS or CAN BUS would let you log and control a range of nodes in an expansive and fully customisable manner.
If its logging, why does it have to be real-time? MQTT is an industry standard for a reason.
EDIT: Daydreaming between uni assignment, daaaayum Jim - biiiiiiig kudos to doing all of this testing! Appreciate you and all of the other regulars on the forum
@Trevor277988 Totally agree with what you are saying.
Interestingly the WiKi page for Private Networks lists a subnet of 255.255.0.0 for 192.186.xxx.xxx. But say it would usually be divided into smaller subnets.
This problem of having a huge number of devices on the same network was why I originally mentioned CISO or something equivalent, router or intelligent switch. And I know the complexity of getting it configured correctly, we has some kick arse routers where I once worked and getting them configured was a nightmare, but once done they just worked nicely.
I was hoping for a cheap option with a Pi as a router, but all I see is setting it up as a WiFi Hotspot. When the idea of using 255.255.0.0 subnet and RaspAP as a bridge occurred to me I saw it as one way @Clem73188 could achieve what he wanted. Cheaply. But it may introduce other problems with so many devices all interconnected. Won’t know till lots of devices are connected and I don’t have that many.
I think it would be well worth @Clem73188 looking at something like @Liam120347 has suggested.
Anyway using a subnet of 255.255.0.0 works for the few devices I have connected. The Pi passes the data through perfectly as it usually does. Very impressed with Raspberry Pi devices.
Agree. Jim is amazing! .When I finish my (first and only) current Pico project, I think I’ll do some playing with a 4/5… and networking. And kudos to Clem for tackling such an complex setup. Cant wait to hear the end of his story!
FWIW… and apologies if you guys are all over this… but the split at octet boundaries, ie 255.255.255.0 vs 255.255.0.0 is entirely arbitrary. No reason you can’t have 10 or 12 bits of host address. 8 and 16 bits are just lazy options. So if you know you will NEVER need more than than 1023 devices, an 11 bit host part would be just fine, ie, netmask 255.255.248.0.
Sub-Networking 101 lesson concluded!
T.
Edit: PS… the above point is moot… unless Clem’s setup actually needs multiple groups of more than 256 devices. Clem, if you do… then the above is highly relevant!
Thanks to all of you guys who have contributed to this discussion - I will reply to all of you and make a communication map of what I am proposing and post it in a few days. And I may reveal some other information about this project that might be of interest. I want to use low cost flexible hardware like the Pi and Pico’s. I also do not want to be locked to general purpose software which typically is written is such a general way and to achieve that there are lots of extra complexities that make the software bulky and much slower than it should be. Maybe I will write my own special purpose router software to run fast on the Pi, I think it cannot be very dificult once the low level issues are understood and sorted. I refuse to confine myself to existing standards like MODBUS which I find discusting in the degraded use of communication just to make it more general purpose. I will be sending data in binary format not converting it to text human readable forms, nor will I be wasting data using 4 bits of a byte instead of all of it. From my nuts and bolts point of view this project is fairly simple - I do not understand the low level interface involved in WiFi hotspots and both at the Pi end and the embedded PicoW end, but I am sure that it involves MAC address and IP addresses plus overheads etc. I could also put another ethernet port on the Pi so that it could feed a switch to give ethernet connectivity inside the containers for some modules which are better interfaced with direct wiring. I want each container to have its own class C network ( SubNetMask 255, 255, 255,0 or as I would prefer to write FF:FF:FF:00) delivered by the WiFi and also the base ethernet network has to be (FF:FF:FF:00) so for a device to connect to the PicoW’s there has to be a bridge because they are strictly on diferent networks. The intention is that the PicoW’s are closed loop controllers and receive their setpoints from a main controller / logger on the base network, therefore this is real time data, not time delayed data. Also this is the reason for UDP packets which contain no stack assembling error recovery / re transmission protocols / overheads, if a packet is lost it is lost, will only give the system a small hickup, but we do not want to loose packets unnescerially. I cannot envisage a need for more than 200 PicoW’s on any one Pi router ( in a container). I will post again when I have made the diagram.
Regards
Clem.
@Clem73188 Thanks for your post, there is a lot to consider.
My investigation has shown the Pi OS is probably not going to do what you want; it is designed for internet access with firewall protection and therefore has network access restrictions. UDP packets have been used by hackers to infiltrate networks in the past.
But, the hardware of the Pi 4/5 would definitely do what you want and you could add USB Ethernet or WiFi dongles.
Having an open subnet mask with an upstream router controlling things for so many devices would lead to its own problems. It would have a massive reserved address table to serve all the static IP’s. And its routing tasks might overload it and slow things down.
Separate C class networks for each container is the best option. Each Pi 4/5 serves the static IP’s for its own network and provides the routing for its own network.
In my test, the PicoW’s were running Micropython and used libraries to connect via WiFi and to send and receive UDP packets. These libraries may have been written in C and incorporated into Micropython, I don’t know for sure. I do know some manufacturers provide drivers for micropython written in C because it works better for the device. This is probably the case with the Infineon CYW43439 WiFi chip used on the PicoW’s.
OpenWRT would be something to look at for the Pi 4/5. It is an open source OS that allows installation of the packages you need for the Pi to do what you want. Of course it might not be the answer. I have yet to look at it. If I do I will let you know what I find. https://openwrt.org/toh/raspberry_pi_foundation/raspberry_pi
Using an already developed product is the best path in any of this, as long as it does what you want. Many years ago I decided to write my own operating system for an Ohio Superboard as the software that came with it did not perform as I would have liked. I quickly realised I would never get anything usable in my lifetime so I had to accept the limitations of the existing system.
Anyway, all the best, look forward to your next post. Happy to provide any assistance I can.
Cheers
Jim
@Clem73188 I new the answer to this would be simple, I just had to get my head around the terminology and how packets are sent and received and how routing works.
But first, I looked at OpenWRT, but after breaking my local net and upsetting those who absolutely need WiFi and internet access all the time, wife; decided it only adds a level of complexity not needed it your case.
My setup now.
Pi 4B running RaspAP. WiFi interface (10.3.141.1) provides an Access Point for PicoW’s and assigns static IP addresses. IP could be whatever you want, subnet mask 255.255.255.0 Ethernet interface (192.168.1.205) static IP assignment by my internet modem router. Provides access to internet and other devices on local network.
PicoW (10.3.141.10) sending UDP packets to another PicoW (10.3.141.12). Testing reliability and speed. Second PicoW is yet to be set up, but the first is running fine, continually sending packets.
Second Pi 4B (192.168.1.203) running latest Pi OS with Gui. Here’s the simple part I was looking for.
Added an IP route so it would forward packets for 10.3.141.0/24 to 192.168.1.205.
Now I can ping 10.3.141.0/24 devices from this Pi 4B. Remove the route and I cannot just like my desktop PC. For my PC to ping I would need to add a route to it.
To Do.
Setup the second PicoW.
Get the second Pi 4B to send UDP packets to both PicoW’s.
Increase the transmission rate of these packets.
Get the PicoW’s to send UDP packets to the Pi 4B.
Collect some statistics on it all.
@Clem73188 Some initial testing on sending UDP packets. Two PicoW’s on same WiFi network.
PicoW 01 sends message number in UDP packet. PicoW 02 listens for UDP packet. Records total number of received packets and error packets.
Packets are very small (less than 10 data bytes) and sent every 3 seconds. This rate will be increased as I get confidence it is working ok. Will also setup two way traffic, ie send and receive to each other. This is using Micropython so it might improve using C or might not, the speed limitation might be the WiFi chip. Will also eventually set it up for the Pi to send to the PicoW’s across the network. The WiFi was using 2.4GHz which is slower than 5GHz. Not sure if the PicoW can connect on 5GHz.
Between 500us and 550us for each send.
Allow you to do some calculations on data rates with a larger number of devices.
@Clem73188 Here are some initial speed tests which you probably won’t like.
2859 packets per sec. 93% error rate. As fast as it will run, no time to do other stuff in mico.
682 packets per sec. 17% error rate. Delay in loop 1ms. Allow time for micro to do other stuff.
95 packets per sec. 2% error rate. Delay in loop 10ms.
23 packets per sec. 0.2% error rate. Delay in loop 100ms.
I have a Pi4B (192.168.1.203) on my local network sending UDP packets to two PicoW’s. The Pico’s reply to the Pi4B every minute or so. Local network is 192.168.1.0/24, Pico network is 10.3.141.0/24. A second Pi4B (192.168.1.205) manages the connection between the local network and the WiFi for the Pico’s.
The trick was to set a route on the Pi4B (192.168.1.203) to send packets for the Pico network to the second Pi4B. So it could act as a gateway. sudo ip route add 10.3.141.0/24 via 192.168.1.205 dev eth0 proto static
If you want more info happy to provide and to help you set it up when it comes to it. But I think this answers your original post to a certain extent.
Totally agree with this.
I was looking at what @Clem73188 had said about how many PicoW’s would be connected to a Pi4B in each container and how much data he would be sending. I have no idea what he wants to do but I have proven a part of what he wanted. Sending and receiving UDP packets by the PicoW’s and having them on separate networks.
Something occurred to me after getting the PicoW’s setup to send and receive. They are basically microcontrollers, they do one thing at a time very well. You can using Python Asyncio to make it look like they are doing two things at once, or more. But each Asyncio task has to have a small sleep time even if only a few milliseconds so the other tasks get some time to run. I discovered this when first using Asyncio. Without await asyncio.sleep(xx) in every Asyncio task the others will not run. You must also use non blocking operations too.
The Pi4B does not suffer from this limitation. It is a microcomputer; it has a full Linux operating system with a task manager. Setting up a separate thread with a blocking operation will work and everything else will work too. I noticed this when sending and receiving UDP packets. The Pi4B handled it much better than the PicoW’s.
@Clem73188 you may find the PicoW’s are not up to the task you want to do. Suggest maybe looking at Pi Zero’s, they are about as small, cost about the same and use the same OS as the Pi4B. Most likely they will be faster and more reliable when connected to WiFi as well as sending & receiving UDP packets. Especially if you look at using the Pi Zero 2 version. Then there are all the versions of the ESP chip set to consider too.
Anyway, just my thoughts. I have the setup running very well at this time given the PicoW limitations.
@James46717 I totally agree with ESP route. On some of my designs I use a LilyGo board that has a normal RP2040 along side an ESP32-C3 running ESP-AT.
Fire up a UART, send AT commands to connect to WiFi, up the UART baud rate to 2M and then put in pass through mode for UDP traffic. No need for asyncio, just smash out the data to the serial port, simplex only.
I use this to communicate Roverling telemetry data to my desktop when it’s in WiFi range, rather than the slower and reduced data LoRa channel. I use this method to transfer 160B every 50ms with no reliability issues. I’ve also tried TCP but the layer 3 overhead and latency is a throughput killer.
Once setup the ESP does all the work of reconnecting and reestablishing the Wi-Fi link if necessary. I’ve also used the MQTT ESP-AT commands with reliable results but at much slower speeds.
@MarkMakies All good information for @Clem73188 to consider. My effort was to give him an idea if it was feasible with what he originally proposed. It was also a learning experience for me.
The result is that it would be feasible with Pi 4/5’s & PicoW’s and WiFi but with some limitations. Without knowing more of what he needs to do it is hard to decide if the Pi’s and WiFi is really what he should consider. I have just read a few of his posts and I think it could be achieved without WiFi and network complexity; just direct cabling to basic micro’s like the ATMega328P.
As he says each setup is inside a converted freight container. The transmission distance would be very short so something like RS485 multidrop would suffice. Each micro is an address on the RS485 bus and the nature of RS485 makes it very robust. Essentially UART serial comms as you have suggested.
Anyway, my testing has concluded and I will not be adding any more to this conversation without some direction. I enjoyed working through the various problems I encountered and gained a better insight into networks as well.
