Orientation Tracking

[Pixmusix]

Happy Monday

I’d love some help dreaming up some kind of that meets this narrative.

brief

Suppose some sensor that points at a rectangular wall and I want to assign (x,y) co-ords to where It’s pointing relative to that wall.

For instance, if I’m pointing into the center of that wall i should my co-ords are 0.5, 0.5 (where 0,0 is the top left corner and 0,1 is the bottom right corner.

Another example, if I’m pointing outside the bounds of the wall I expect a valid reading, i.e. if I point left of the wall I want x<0 and if below the wall y>1

my first guess

I want something very robust and instantaneous so I think embedded systems AI is too slow.

I’m imaging some emitters (maybe infrared or similar) equally spaced on the wall each with a signature that allows them to be differentiated (each with a different frequency for example).

Using my sensor to track distance/return-time from each emitter I can project the location I’m pointing to with some trig. Here is an MS paint sketch.

restrictions

I want the final sensor to be able to be handled by the public. That means no lasers, nothing that can blind someone, less than 48v, absolutely no exploding etc. (given the graveyard of capacitors I’ve been building up maybe that’s a universal boundary I should hold for myself too).

Anyone got a hardware suggestion?
Better plans? (love me better plans).

Thanks in advance
Pix

[Jeff105671]

Can the sensor communicate with the wall apart from the sensing (eg, ESPNOW)? If so you could use a wall full of IR emitters that are pulsed in turn, with a timing start pulse sent to the sensor. When the sensor detects the IR blink it knows which emitter the pulse came from by the timing delay from the synchronizing signal.

Or, how about a wall full of IR emitters each operating at a different frequency? The detector has a shielding arrangement that keeps the FoV to a narrow segment, and reads the frequency of the received IR beam. Left/right/top bottom get their own frequencies from multiple emitters over a larger area. Off-screen corner areas are currently undefined.

IR emitters are cheap and you could get quite a lot of frequencies for them with just a handful of pulse generators with digital dividers. You would need to confirm the frequency measurement discrimination that was possible. Overlapped regions would be a problem - could be handled as a null area depending on the application or processed using FFT if there is the processor grunt.

[Jaryd]

This is a deceptively tricky project you have going on here and seems like it can very quickly run down some deep mechatronic rabbit holes.

My initial thought would be an IMU to get the orientation of the pointing device and use it with a laser distance sensor to figure out where it’s pointing. But the second you move closer to the wall or side to side it would throw everything out of whack. Figuring out that extra translational dimension becomes another challenge in of itself with something like wideband.

I am gonna phone in a friend here who might be able to help @Liam120347.

[Liam120347]

Hi Jonny,

Jaryd is very right that it quickly turns into a mechatronic project.

I like Jeffs idea, and haven’t seen it done before but would be very interesting!

If you just need the direction its pointing it will be much much easier

I would take one from the books of a commercial product, the Wii - I came across this reddit post with a couple of handy links: Reddit - Dive into anything

Regardless, if you want to go down the rabbit hole, the system will have to do some sort of sensor fusion.

Not having to get the distance between anything makes this so so so so so so so much easier.

Another thought,
How many features would be on the wall? Like Jeffs idea, could you put one of these emitters inside of one of the objects, then put a mask around the pointing device so it can only see one of the features at a time?

I’m not too sure how robust getting the ToF from sensors off the wall might be, Jeffs solution gets around this by making each requested pulse a constant offset - Bob has mentioned this a times over the years to similarish projects (but they are usually more concerned about position).

An insight into industry

At uni we have access to an IR multicamera optical tracking system - Vicon, that allows for high refresh sub-mm accuracy tracking for objects

I’ve also been meaning to get my hands on some ultra-wideband modules for indoor GPS

Regardless of the technology the position measurements are only made possible through sensor fusion

[James46717]

This reminds me of an arcade game where you point a rifle at an object on the screen and pull the trigger. The screen flashes various points in sequence faster than the human eye can recognise. The gun has a sensor that responds when it picks up a flash, based on the timing of flashes and the pull of the trigger the system then knows which area the gun is pointing at. The sensor in the gun is deep within the barrel and will only see a small part of the screen.

But this is like @Jeff105671 idea of many IR sensors on different frequencies. To do it with only 3 emitters I don’t really see any easy options. TOF of the frequency for each emitter would then need some sensor fusion stuff. Probably similar to how a GPS sensor works out where it is by the timing of signals from a number of GPS satellites.

But that is very different from what you are proposing. The sensor could be pointing at the same spot but in a different location so the TOF would be different. I think it could be done but it would take some fancy software well beyond my skill level and understanding.

Regards
Jim

[Jeff105671]

I think the three emitters was only an example. If it was restricted to three the problem becomes much more difficult.

I’m not sure if your comment re TOF refers to my ideas or not, but they do not involve TOF. I was expanding on what OP had mentioned, using a pointer like the rifle barrel in your example, so that the FOV of the sensor is small enough to distinguish each emitter. Measure the frequency and you know which emitter is the one in view.

The first problem is that the emittters might swamp the area making it impossible to limit the FOV to a sufficiently small area. The second problem might be that when the FOV is limited to a sufficiently small area the signal might not be strong enough to be readable at the sensor. That would need to be tested in the actual environment.

My other suggestion is a bit more like the lights in your example. It requires the same limit on the FOV so only one emitter is detected, but uses a synchronising procedure. That could be by wireless (ESPNOW is fast enough to do that) or it could be based on encoding of the pulse train, like a UART uses Start and Stop pulses. None of that involves TOF. There would be a practical limit on the number of emitters that could be fired before the pulse timing became unreliable.

[Pixmusix]

Hiya again.

Thanks to everyone who has replied
I’ve been reading this thread over the last days and just letting all the ideas sit with me for a bit to see how I feel about them.

I think what I’m learning reading everyone’s thoughts is that I’m going to either need
a) some kind of complex sensor fusion
b) a compromise to my goals

In reality land the answer is all of the above.
Let’s start with b).

Compromises

I think my affordable, low solder, and elegant plan of triangulation was optimistic. Seems like it’s easy to detect “that” I’m looking at an IR emitter but not easy to gain metadata (like distance to) on that emitter.

I think that means It’s not realistic to expect information when pointing outside the bounds of the wall. i.e. If I can’t receive information from any emitter, my vector is undefined.

I think I have too many degrees of freedom so I’m going to have to decide on, measure, and lock in a distance.

With those compromises made I’d like to restate my problem.

Suppose a detector 1 meter from a 1 by 2 meter featureless blank rectangle (wall).
How can I find the x,y coordinates of the point my detector is looking at on that wall.

Responding to your ideas.

@Jeff105671’s IR’s at different frequency with low FOV

I think this is clever but it’s not my favorite. Sounds a little finicky and I can’t really find a device that can detect IR at different wave lengths.
For example this one can detect IR but only at 940nm. This guy too but only at 950nm (and I’m not even sure that’s enough of a difference to be reliably distinguished.

@Jeff105671’s IR cylon IR emitters.

I feel this is super promising. I built what amounts to an led-follower in a recent project and I think I can use a lot of the same techniques here. Synchronization is the hiccup but ESP_NOW would work i bet. Never used ESP_NOW but having glanced at the website it doesn’t look scary.

Thinking it through I think I can avoid a wireless communication protocol entirely.
I don’t see why I can’t just use two clock sources of the same frequency that just occasionally sync. (a bit like RS-232).

For instance, suppose the 16 IRs in a 4x4 gird. In each cycle they each pulse in turn starting top left to bottom right. So like …

0 1 2 3
4 5 6 7
8 9 A B
C D E F

What I’m proposing is that at the beginning of each cycle I just flash ALL the IRs. With slow enough clocks I wonder whether that’s enough to keep two independent clocks together without a wireless protocol.

I suppose hardware matters here because my idea only works if my detector can distinguish 16 leds from 1 BIG led. This detector looks interesting but I don’t know what wave lengths it can detect, (or if it can tell the difference between them).

Positioning IR Camera

Testing required? Any critiques?

Sensor fusion with @Liam120347 & @Jaryd

I made a compromise to fix my distance but that means a gyroscope would be a perfectly reasonable way to keep track of the vector I’m pointing. I think hobby gyros slip a little fast but if I fuse it with some IR technique I might be able to use less emitters because I can continuously re-calibrate with each successful IR reading.

Gear.

I’m struggling to find what I think I want in CE’s inventory et al. I guess I’m looking for Wii remote tech but in 2024. What does that look like? Is that Positioning IR Camera in 2024 the right thing?

What’s the best Gyroscope? I know you often find them in accelerometers but something that’s quality and dedicated to rotation sounds more like what I’m after. I don’t really want to muck around with xyz movement if … ya know… I’m not moving.

I guess this emitter will work fine.

Thanks everyone for work-shopping this with me.

[Jeff105671]

Your initial idea of IR emitters at different frequencies does not involve different wavelengths. The IR emitters are all identical and all emit at the same wavelength and the sensor in the wand responds to them all in exactly the same way. What distinguishes the emitters is the rate at which they are pulsing, and this is what the sensor is reading. That’s why I mentioned a pulse generator (eg 555) and prescaler/divider (eg 74393). For instance, if you generate a stream of pulses at a frequency of say 3.2kHz and divide that by 4 to drive 4 IR emitters at 1600, 800, 400, 200 Hz. Then another at 4.8kHz to give 2400, 1200, 600, 300. You get 8 distinguishable frequencies for pulsing the emitters from three chips and a handful of static components. A 556 with 7459x reduces the IC count to 2 for 16 different frequencies with slightly less flexibility. The limit is set by the maximum frequency the receiver can reliably count, and the frequency difference required for correct discrimination.

The second possibility is also basically the same as I mentioned. Think of it as a UART transmitting a 16-bit word where only one bit is set. It needs a start pulse to start the timing and synchronize the clocks, the word at one bit per time interval, and a stop bit. But there is a limit to how large a word can be sent, as you note, because there is a reliance on the clocks being in sync. As with a UART, the slower the baud rate the longer that word can be. However, if you have two-channels available then you can sync on each bit (eg, I2C) and the word length doesn’t matter.

[Pixmusix]

@Jeff105671

Your initial idea of IR emitters at different frequencies does not involve different wavelengths.

Ah, I see where we got ourselves confused. I totally did mean different wave lenghts. I was wanting IR at different periods.

@Jeff105671

What distinguishes the emitters is the rate at which they are pulsing, and this is what the sensor is reading

I now understand what you’re talking about! Sure. I get what your saying about a low FOW. With a low FOW I’d need higher density of emitters to compensate. Not sure I’m keen to do that much soldering.

@Jeff105671

Overlapped regions would be a problem - could be handled as a null area depending on the application or processed using FFT if there is the processor grunt.

This sentence, in retrospect, now clicks. I’m not a stranger to FFTs but I’m not sure how much fun implementing one in embedded systems would be. I found this… doesn’t look too bad.

@Jeff105671

It needs a start pulse to start the timing and synchronize the clocks, the word at one bit per time interval, and a stop bit.

Yeah I follow. I’ll start reading up on some different protocols and see what I can dream up.

[Jeff105671]

It’s just occurred to me that the datastream idea can be easily used with a wider FOV. For instance, from the numbering in your diagram, if the data received was
0 0 0 0 0 1 1 0 0 1 1 0 0 0 0 0 then the pointer is centred on the wall. The interpolation process also means improved resolution.

Note that the start pulse is usually identified by the pulse width. An end pulse is not essential, but it could be used (again, identified by the pulse width) as a sanity check on the accuracy of the bit counting.

ESP_NOW is really easy and it has the advantage of being peer-to-peer - no latency from intemediate devices.

[Pixmusix]

@Jeff105671

The interpolation process also means improved resolution.

This is what I’m going for above
But, because of the wider FOV, what worries me is whether or not sensor will be able to handle reading a start pulse (all 16 on at once or 0xFFFF).
Hopefully ce staff can recommend something.

[Jeff105671]

@Pixmusix

whether or not sensor will be able to handle reading a start pulse (all 16 on at once or 0xFFFF).

If all the emitters are issuing the start pulse then the sensor only has to detect any one of them. The start pulse is identified by the pulse length. A reading of 0xff would only be available if a start pulse had already been detected.

[Zach]

Hey @Pixmusix, @Jeff105671,

This forum chain has been really interesting to read through!

I’m happy to assist recommending an IR sensor, though I was hoping to confirm the exact specifications first.

From the sounds of it, the plan is to make an array of IR transmitters on the board that flash at different frequencies (the IR is at the same frequency but the transmitters themselves pulse at some different frequency) to dictate their position on the board, and have an IR receiver placed at a distance of 1m from the board.

Is this correct? If so, I would think any IR receiver would work, given that it’s effectively just an analogue of whatever IR transmitter it’s pointed at.

Let me know if I’ve hit the nail on the head, or missed it completely!

[Richard83832]

Another brute force approach to consider for this very interesting but seemingly difficult project may be to determine the relative position of the pointer to the target and then fuse that data with the absolute orientation of the pointer. You should then be able to get valid data even if the pointer was aimed outside your target area. The relative position may be determined by a pair of RTK GPS receivers if outdoors, or possibly via angle of arrival technology if indoors. Both these technologies are sophisticated and involve software from the manufacturer ublox. The absolute orientation of the pointer is more easily determined by a separate IMU or GPS onboard IMU. Like most projects, your success will probably depend on your budget and performance criteria.

Another much cheaper and simpler approach may be the use of computer vision. Your pointer would contain the computer vison camera and processor (both could be R Pi products) to run say an OpenCV algorithm to accurately determine the camera’s orientation relative to your target board. The coordinate data of the target could be displayed either on the pointer or sent to a remote screen. I believe this approach is more promising in terms of cost, simplicity and accuracy. Let me know if you require further info on these ideas.

[Liam]

Hi Jonny!

On the hardware side I think we’ve all been been alluding to this kind of device: Light gun - Wikipedia

To get you some links I would go with any of the 38KHz IR devices: https://core-electronics.com.au/catalogsearch/result/?q=38khz%20ir

Adafruit Infrared IR Remote Transceiver - STEMMA JST PH 2mm - 940nm Emitter +...

Adafruits Transceiver will let you get x amount of the same device and let you tx and rx if you would like!

As for the camera - apparent the Wii remote’s IR camera had a resolution of 1024x768, and an FoV of 45 deg, compared to a 128x96, and 33 deg horizontal and 23 deg vertical for the DFrobot device.

This solution also comes with quite a bit of math - outside of any sensor fusion!

I reckon Jeff is on the money with the grid of emitters, I think it will be the most cost effective and time effective solution.

And if you are keen on learning sensor fusion it still returns a (lower resolution) discretised state that you could still update with sensor fusion and an imu

@Pixmusix

What’s the best Gyroscope? I know you often find them in accelerometers but something that’s quality and dedicated to rotation sounds more like what I’m after. I don’t really want to muck around with xyz movement if … ya know… I’m not moving.

I’d still go for a 6dof IMU like an MPU6050, the sensor fusion merges the accelerometer and gyro, significantly decreasing the gyro drift.

Keen to see how this one comes out!
Liam

[Pixmusix]

Hi @Zach

What I’m settling on today is cylon-ing emitters with independent clocks @ an pre-agreed frequency. All flash a start bit to begin a sequence, synchronizing the clocks, and then the emitters cylon down the wall like the animation below.

The receiver can figure out what part of the wall it’s looking at by asking “at what count do I see infrared”. If the receiver sees IR only at zero based counts 14, 21, 22, 23, & 30 then it knows it is in the bottom right quadrant.

Hopefully that catches everyone up to where @Jeff105671 and I are at.

[Pixmusix]

@Liam

To get you some links I would go with any of the 38KHz IR devices: https://core-electronics.com.au/catalogsearch/result/?q=38khz%20ir

Thanks @Liam
This below looks interesting. Low FOV and 116cm of range.
It calls itself a temperature sensor.
Is there a meaningful difference between a temperature sensor and an IR sensor or is all just marketing.

TS01_IR_Thermal_Sensor_(0-3V)_SKU_SEN0256-DFRobot

[Robert93820]

@Pixmusix

Suppose a detector 1 meter from a 1 by 2 meter featureless blank rectangle (wall).
How can I find the x,y coordinates of the point my detector is looking at on that wall.

Not too hard.
Firstly establish a datum point as the centre of the board and the “sense” exactly at 90º to that.
You need to know the EXACT distance from the wall to the PIVOT POINT of the “sensor”
Using one of the PiicoDev products which measures a change in angle measure the angles both horizontal and vertical which the pointer has moved.
The actual DISTANCE the pointer has moved from centre can be calculated (Both horizontal and vertical) which will result in the offset from centre in whatever units are used (mm???). The offset is the Tangent of the angle multiplied by the distance from pivot point to board…
The idea here is to have the datum point at the centre of the board and measure the offset distance (H and V) from there. A bit of arithmetic should enable you to have absolute cordinates (X and Y) with the origin wherever you want it.

As I see it the hardest part will be finding Arduino or RPi that will handle Trig functions like Tangents. I am not sure about this as I have never had the need but surely one of these devices will do the job.
Cheers Bob

[Pixmusix]

@Robert93820

The idea here is to have the datum point at the centre of the board and measure the offset distance (H and V) from there. A bit of arithmetic should enable you to have absolute cordinates (X and Y) with the origin wherever you want it.

Thanks Bob. Great thoughts. I figured something like this would need to be included.
I think I’m going for a little bit of sensor fusions between a gyro with an IR receiver. This way I won’t need to worry about gyroscopic drift or establish an absolute datum point.
Hoping that combining what you’re suggesting with what we’ve been work shopping above will be enough. Testing will tell

[James46717]

@Robert93820

finding Arduino or RPi that will handle Trig functions like Tangents

@Robert93820 RPi will have no problem handling Trig functions. Many libraries exist that will work. The Arduino might struggle a bit.

Reading all this with interest I had a thought just now. Two cameras on the board and an AI object recognition system might work, it could detect location and distance, then zoom to work out what angle the pointer is at; maybe this is pie in the sky because I don’t know enough about AI object processing at this stage.

Regards
Jim

PS For now what you are looking at is well on the way @Pixmusix