Crazyflie logged state estimates freeze, but log timestamps are updated (+Lighthouse)

[tiagomsr99]

Hi, I'm currently controlling two Crazyflies using a Matlab/Simulink controller.

I've a Python script that uses the Swarm framework with a simple loop: receive state from the Crazyflie -> send it to Matlab/Simulink through UDP -> receive the Matlab/Simulink controller output -> send the control command to the Crazyflie using send_setpoint(roll, pitch, yawrate, thrust). As for the Matlab/Simulink, I've both controllers on the same file, the only thing that changes is the ports necessary for the UDP connection.

One drone works well (drone1), the other (drone2) starts well but then 1.8s later the state estimates freezes: https://i.imgur.com/cEcIvj3.png
Which is weird as the log timestamps arrive correctly: https://i.imgur.com/DSX8pBO.png
Here's the flight data: https://i.imgur.com/wJAmuLy.png
And a video of the occurrence: https://imgur.com/a/3dB8F9d

According to the video, drone2 drifts to the left (positive y in my setup) and straight (positive x in my setup) while slowly increasing the height. This can't be seen in the flight data because the state estimate freezes at
x = -1.3252, y=-1.0146, z=-1.5684

As the controller keeps working, and believes drone2 is at those positions, the left-straight drift makes sense as it's trying to go to the desired reference:
x = -0.33, y=-0.33, z=0.4

The UDP network runs at 2.4GHz, I'm logging at 50ms and here's the URIs:
'radio://0/80/2M/E7E7E7E7E7
'radio://0/80/2M/E7E7E7E701'

Both Crazyflies have the most recent firmware, no parameters or flags changed, and I'm using the Lighthouse positioning system with 2 basestations. I can control each drone individually with no problem.

I've observed this drifting behaviour in other flight tests when the Lighthouse loses sight of the Crazyflie. When this happens, position values tend to minus infinity, but this is the first time this happens and the log state estimate freeze occurs. I've noticed that when the Crazyflies are not fully charged this Lighthouse related drift happens a lot, I've read somewhere that It's a known bug related to the Lighthouse deck power?

I find it hard to believe that the Lighthouse can't detect the Crazyflie in that location, as I've tested other trajectories and scripts with no problem (for instance, autonomous_sequence_high_level.py).

So I think I have two problems: the Lighthouse-Crazyflie connections fails (happened to me in other flights, explains weird values such as negative height) and then the state estimate freezes.

• How can I prevent or what causes the Lighthouse to lose connection to the Crazyflie? What is the best Lighthouse log variable to debug this connection issue?
• What may be causing the state estimate to freeze?

I'm not using

def reset_estimator(cf):
    cf.param.set_value('kalman.resetEstimation', '1')
    time.sleep(0.1)
    cf.param.set_value('kalman.resetEstimation', '0')

    wait_for_position_estimator(cf)

as I've had good single Crazyflie flights without it, should I use it and can this be causing my problem?

Thanks in advance!

EDIT:
I'm logging this, in a single log block and for each Crazyflie at 50ms:

self.log_config.add_variable('stateEstimate.x', 'FP16')  
self.log_config.add_variable('stateEstimate.y', 'FP16')
self.log_config.add_variable('stateEstimate.z', 'FP16')
self.log_config.add_variable('stateEstimate.roll', 'FP16')         
self.log_config.add_variable('stateEstimate.pitch', 'FP16')
self.log_config.add_variable('stateEstimate.yaw', 'FP16')
self.log_config.add_variable('stateEstimate.vx', 'FP16')    
self.log_config.add_variable('stateEstimate.vy', 'FP16')    
self.log_config.add_variable('stateEstimate.vz', 'FP16')
self.log_config.add_variable('gyro.x', 'FP16')     
self.log_config.add_variable('gyro.y', 'FP16') 
self.log_config.add_variable('gyro.z', 'FP16') 

[knmcguire]

Hi! Let's try to debug your problem.

• Probably good to look at raw angle receivings: https://www.bitcraze.io/documentation/repository/crazyflie-firmware/master/api/logs/#lighthouserawangle0xlh2
• Have you tried switching the lighthouse decks? Are you seeing the same problem then?
• Have you tried switching the male pins? or tried a different battery?
• I see that once the crazyflie takes off, the lighthouse deck seems to change from green to orange, is that correct?

[tiagomsr99]

Switching the Lighthouse decks: the problem persists. I've noticed a slightly crooked male pin, replaced it, but changing it made no difference.

I've also switched both drone's starting position, to test if that specific flight area had bad connection to the Lighthouse, and the same behaviour occured so there's definitely something wrong with Drone2.

Then I've switched drone2's battery with a new one and it worked! Well that was an underwhelming debug, but thankfully an easy one to solve.

Thank you so much!!

[knmcguire]

Ah interesting! I mentioned it since I've observed something similar in our lab too and that the fix was a new battery. We need to figure out why that is though...

Just for the sack of sanity, could you perhaps check the quality of the bad versus the good battery with a battery health check? Just so that we can compare.

[knmcguire]

btw I selected your reply above as answer for this discussion as that mentions the battery being the issue :)