discussing about the I2C hanging issues

[jerabaul29]

This is moslty just for discussing / brainstorming.

There is this really annoying issue with the BNO080 hanging when put on I2C to the Artemis.

Any feedback you received about connecting the BNO080 through UART or SPI instead? Does that work without hanging? Of course I would prefer I2C, but if needed, I can use something else.

It is very late, will try tomorrow, but would be great to know if there are some known issues there too / if investigating this would waste my time. In this case, I would stick to I2C and the hard reset solution ^^ .

[jerabaul29]

I closed issues #68 and #69 in favor of this discussion.

I spent quite a bit of time investigating these issues, so I thought I would give a small update here. That may be useful for other users who hit the same issues.

Because of some issues with querying several data feeds with the current SF code (see #70 ), I use the Adafruit code. There is a bit of a hack to get it to work on the Artemis board, see the recipe https://github.com/jerabaul29/Artemis_MbedOS_recipes/blob/main/recipes/recipe_IMU_BNO_Adafruit/recipe_IMU_BNO_Adafruit.ino comment at the start to explain how to get it to work.

I use this script to follow the advices of @PaulZC and restart the board if connection fails (of course the issue is that I then loose all the RAM information; a solution would be to dump user data to Flash through the "EEPROM" library):

#include "Arduino.h"
#include "WDT.h"

#include <Adafruit_BNO08x.h>

Adafruit_BNO08x bno08x{};
sh2_SensorValue_t sensorValue;

unsigned long micros_last;
unsigned long micros_crrt;

void setReports(void) {
  if (!bno08x.enableReport(SH2_ROTATION_VECTOR, 500000UL)) {
    Serial.println("Could not enable rotation vector");
  }
}

APM3_WDT wdt;

void resetArtemis(void)
{
  Wire.end(); //Power down I2C

  SPI.end(); //Power down SPI

  power_adc_disable(); //Power down ADC. It it started by default before setup().

  Serial.end(); //Power down UART

  //Force the peripherals off
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM0);
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM1);
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM2);
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM3);
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM4);
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM5);
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_ADC);
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_UART0);
  am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_UART1);

  while (1) // That's all folks! Artemis will watchdog reset in 1.25 seconds
    ;
}

void setup(void) {
  wdt.start();
  
  Serial.begin(115200);
  while (!Serial){
    delay(10);
  }

  Serial.println();
  Serial.println("------ booted ------");
  Serial.println();

  // initialize
  for (int i=0; i<60;i++){
    wdt.restart();
    if (bno08x.begin_I2C(0x4B, &Wire, 0)){
      Serial.println(F("found the Artemis!"));
      break;
    }
    else{
      Serial.print("Failed to find BNO08x chip "); Serial.println(i+1);
      if (i == 19){
        Serial.println(F("reset Artemis"));
        resetArtemis();
      }
      Wire.end();
      delay(50);
      Wire.begin();
      delay(50);
      
      delay(500);
    }
  }

  Wire.setClock(400000);
  delay(50);
  
  setReports();

  micros_last = micros();

  Serial.println("----- Reading events -----");
  delay(100);
}

void loop() {
  Serial.println(F("start reading"));
  
  while(true){
    if (bno08x.wasReset()) {
      Serial.print("sensor was reset ");
      setReports();
      continue;
    }
  
    if (!bno08x.getSensorEvent(&sensorValue)) {
      continue;
    }
    else{
      wdt.restart();
          
      switch (sensorValue.sensorId) {
      case SH2_ROTATION_VECTOR:
        micros_crrt = micros();
        Serial.print(F("micros: ")); Serial.print(micros_crrt); Serial.println(F(" | micros elapsed: ")); Serial.println(micros_crrt-micros_last);
        micros_last = micros_crrt;
        
        Serial.print("Rotation Vector - r: ");
        Serial.print(sensorValue.un.rotationVector.real);
        Serial.print(" i: ");
        Serial.print(sensorValue.un.rotationVector.i);
        Serial.print(" j: ");
        Serial.print(sensorValue.un.rotationVector.j);
        Serial.print(" k: ");
        Serial.println(sensorValue.un.rotationVector.k);

        // TODO: may want to put a small delay here
        
        break;
      }
    }
  }
}

I noticed a few interesting things. I think there may be 2 different bugs that cause hanging. Indeed, I get 2 "flavors" of hanging.

• One hanging jumps to the Failed to find BNO08x chip message. This issue is of course a bit annoying, but it is benign in practice in my tests: by looping on attempting to connect to the BNO, I usually manage in the end to talk to it. A funny thing here is that there seems to be some form of regularity. For example, if 1) I connect the board, and it goes through the cycles of re-trying until connecting and getting good data. Then if 2) I press the reset (or rather reboot) hardware button on the Artemis, then 3) I have observed "experimentally" that often the same hanging happens, and usually (not always) as many retries are needed each time.

• Another issue, which is much more damaging to me, is that it seems that sometimes the boards hangs altogether. In this case, I do not even get the Failed to find BNO08x chip chip, and the code hangs in the bno08x.begin_I2C(0x4B, &Wire, 0) function. This is much more problematic, as the whole thing seems to then freeze, and in this case I do need the watchdog reset and I loose the data.

[jerabaul29]

@PaulZC I was thinking more about that. Any way that we could restart the I2C port, i.e. power it up and down, from the watchdog, without the need to restart the whole board? That may be a workaround to reset the port without loosing the data in RAM, right? Any idea this could be doable / is there a HAL call for that?

[jerabaul29]

I am trying to reset the I2C hardware from within the interrupt with something like this:

volatile bool force_wait {true};  // no delay() in interrupt; force reading and writing a volatile variable
volatile int remaining_watchdog_tries {10};

// Interrupt handler for the watchdog
extern "C" void am_watchdog_isr(void)
{
  wdt.clear();
  
  if (remaining_watchdog_tries > 0){
    wdt.restart();
    remaining_watchdog_tries -= 1;
    Serial.print(F("retry I2C "));
    
      Wire.end();
      am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM0);

      // hand-made delay, to make sure power has time to go down
      for (int i=0; i<1000000; i++){
        force_wait = !force_wait;
      }
      
      am_hal_pwrctrl_periph_enable(AM_HAL_PWRCTRL_PERIPH_IOM0);
      Wire.begin();
    
  }
  else{
    // reset
    while (true){;}
  }
}

[jerabaul29]

I even tried to disable and enable all the periphs (IOM0 to end), but that does not help.

I do not know / have no way to know as far as I know (some method to get this information?) where the code hangs. I would say that now in around 90% of the cases I manage with the looping to get contact with the BNO, and in around 1 in 10, the code hangs when trying to begin the BNO, the watchdog kicks in, tries 10 times to restart the I2C, but with no success, leading to a full reset.

If there was a way to trigger a reset of the right I2C stuff from the watchdog, rather than by doing a full chip reset, that would practically solve the problem. I do not have the JTAG or similar debugging tool nor the experience to debug down to this level / get this to work. If you have any advices @PaulZC that would be amazing.

[PaulZC]

Hi JR,
Like I said in this comment, I have already spent hours looking at this. But, I must admit, I haven't tried JTAG. That might be a way forward - but I really don't have time to look at it right now.
If you want to dig into it, start looking in here:
https://github.com/sparkfun/Arduino_Apollo3/blob/v1/cores/arduino/ard_sup/iomaster/ap3_iomaster.c#L759
That function is used for both I2C and SPI - not just SPI.
If I recall correctly, the behavior is that the hardware gets into a state where am_hal_flash_delay_status_check always times out, preventing any further I2C transactions and causing the bus to hang.
If you solve this, I definitely owe you a beer!
Paul

[jerabaul29]

@PaulZC just FYI: it looks like I "empirically" found a recipe that works all the time at starting the BNO with Artemis, at least on the Artemis RedBoard and AGT I have here for testing, and with the couple of BNOs I have here too ("works on my computer" symptom).

This recipe:

https://github.com/jerabaul29/Artemis_MbedOS_recipes/blob/main/recipes/recipe_IMU_quaternions/recipe_IMU_quaternions.ino

With in particular this snippet:

https://github.com/jerabaul29/Artemis_MbedOS_recipes/blob/5984c4c799a52c4979392e0e71386a56da78fb3a/recipes/recipe_IMU_quaternions/recipe_IMU_quaternions.ino#L36-L67

Has been working without any issues now for me several tens of times. Other similar snippets with different timing or checks fail regularly, but just this one seems to work.

I would be interested to hear if other users are able to use this snippet to get a reliable startup.

[jerabaul29]

(you really do not want to know how many hours I spent having trial and errors to get to this one...).

[PaulZC]

Hi JR,
Sorry, I should have mentioned that earlier. I did notice that calling Wire.setClock(400000); before attempting to start the BNO improved the performance. Why should 400k be more reliable than 100k? I really have no idea. In my mind it just confirms that there is something 'marginal' about the combination of the Artemis hardware with the BNO.
Best wishes,
Paul

[jerabaul29]

Yes, there is something really fishy. I have been "test trying" for around 1 hours to get this snippet to crash in the same way other similar snippets do, "luckily without" success. I think there is a broken piece of code somewhere and that somehow having just this combination of delays allows to avoid it.

[jerabaul29]

For the records, Teensy forum has a loooong discussion around that:

https://forum.pjrc.com/threads/58268-Sparkfun-BNO080-Teensy-3-6-gt-gt-4-0

Read through it now, and not completely sure what the conclusion was ^^ .

[adamgarbo]

Hi folks,

I just received my SparkFun BNO080 boards yesterday and spent a couple of hours trying to troubleshoot why they wouldn't initialize with Artemis-based boards before I found these issues on GitHub.

I'm still catching up, but it sounds like the main takeaway is that the Artemis and BNO080 are, for the moment, incompatible over I2C?

Thanks, @jerabaul29 and @PaulZC for your work in looking into this.

Cheers,
Adam

[jerabaul29]

Hi @adamgarbo ,

Yes, had the same kind of troubles unfortunately :( .

A few notes:

• with a snippet in the kind of the previous one (actually a bit longer, see below), I was having a good success rate at waking up the BNO and connecting to it. Together with a watchdog reset it is quite ok to use. The only problem is the loss of RAM data at reset, but this could be solved by using the "EEPROM" stuff for holding important data I suppose :) . I have 2 intruments in the middle of the Arctic using this solution, together with the quaternion analysis similar to what is done in https://github.com/jerabaul29/Artemis_MbedOS_recipes/tree/main/recipes/recipe_IMU_quaternions , and they seem to work nicely.

• I found out that the RVC (Robot Vacuum Cleaner) mode, using UART at baud 115200, works perfectly and is very reliable. I have a recipe here (it was on the Artemis Global Tracker, so I had to trick a bit and use a hardware serial to talk to the BNO as it is quite fast, and reconfigure the serial to the Iridium as a software serial since the hardware serial got used, but this is fine, as software serial works very well at the baudrate 9600 of the Iridium modem). See: https://github.com/jerabaul29/Artemis_MbedOS_recipes/blob/main/recipes/uart_rvc_bno08x/uart_rvc_bno08x.ino . I have 12 instruments in the middle of the Arctic using this solution, and it seems to work perfectly.

So my advice would be, if it is ok for your application, to use the RVC mode. As far as I found out, the only negative points for my use were that:

• it outputs Euler angles and not quaternion, which is much less satisfactory from a mathematical point of view, and also, seem to have a gimbal lock when tilt is around 90 degrees. For my (and possibly your I guess :) ) applications, this is not a problem though.

• output is fixed at 100Hz. This is fine for me though.

• the orientation is given relative to the direction at startup, and not relative to the North, and I do not think that there is a way to get information about heading relative to the North. For my current application this is not an issue, but for later application, this will become a big issue from my side. :(

So it is in general quite ok to use, but it will make for a good story in my department, when I explain to my boss and colleagues that our instruments believe that they are robot cleaners ^^ .

These problems are very sad, because this is an amazing sensor. I would be very curious @PaulZC to know if using the BNO085 could fix this issue, and if Sparkfun would be ok shifting to these in the future :) .

PS: my current snippet for the I2C. Ugly, ad-hoc, hacky, but seems to do the trick around 90% of the time in my tests:

bool BNO080_Manager::_power_start(void){
  turn_bno_off();
  delay(1000);

  Wire1.begin();
  delay(50);
  Wire1.setClock(400000);
  delay(50);
  wdt.restart();

  delay(2000);
  wdt.restart();
  turn_bno_on();
  delay(5000);
  wdt.restart();

  bno080_imu.begin(BNO080_DEFAULT_ADDRESS, Wire1);
  delay(50);
  bno080_imu.enableDebugging(Serial);
  delay(50);
  wdt.restart();

  bno080_imu.enableRotationVector(imu_output_period);
  // bno080_imu.enableAccelerometer(imu_output_period);
  wdt.restart();

  delay(5000);
  wdt.restart();

  while (!bno080_imu.begin(BNO080_DEFAULT_ADDRESS, Wire1)){
    Serial.println(F("try to begin"));
    delay(100);
  }

  delay(5000);
  bno080_imu.begin(BNO080_DEFAULT_ADDRESS, Wire1);

  wdt.restart();

  Serial.println(F("bno set up, start measuring"));
  wdt.restart();
  
  return true;
}

bool BNO080_Manager::_first_measurement_start(void){
  //--------------------------------------------------
  Serial.println(F("wait for the first BNO output"));

  bool got_first_reading {false};

  // TODO: protect, this can hang
  for (int i=0; i<15;i++){
    if (bno080_imu.getReadings() != 0){
      got_first_reading = true;
      break;
    }
    Serial.println(F("receiving nothing, try to set output"));
    bno080_imu.enableRotationVector(imu_output_period);
    delay(50);
    bno080_imu.enableAccelerometer(imu_output_period);
    delay(50);
  }
  wdt.restart();

  if (!got_first_reading){
    Serial.println(F("could not get a reading"));
    return false;
  }

  // bno080_imu.enableAccelerometer(imu_output_period);
  bno080_imu.enableRotationVector(imu_output_period);
  wdt.restart();

  Serial.println(F("got first measurement, check has both quat and accel"));

  //--------------------------------------------------
  // check that we can get both a quat and an accel reading
  delay(100);

  Serial.println(F("get accel and quat"));

  while (true){
    if (bno080_imu.getReadings() == SENSOR_REPORTID_ACCELEROMETER){
      Serial.println("accel ok");
      break;
    }
    delay(1);
    // bno080_imu.enableAccelerometer(imu_output_period);
  }

  while (true){
    if (bno080_imu.getReadings() == SENSOR_REPORTID_ROTATION_VECTOR){
      Serial.println(F("quat ok"));
      break;
    }
    delay(1);
    bno080_imu.enableRotationVector(imu_output_period/2);
  }

  //--------------------------------------------------
  Serial.println(F("ignore first BNO measurements"));
  Serial.flush();
  delay(5000);
  wdt.restart();

  for (int i=0; i<50; i++){
    while (true){
      if (_perform_one_reading() != 0){
        break;
      }
      delay(1);
      wdt.restart();
    }
  }
  wdt.restart();

  Serial.println(F("ready"));
  Serial.flush();
  
  return true;
}

bool BNO080_Manager::start(void){
  bool start_succes {false};

  for (int i=0; i<5; i++){
    if (!_power_start()){
      stop();
      delay(1000);
      continue;
    };
    wdt.restart();

    if (_first_measurement_start()){
      start_succes = true;
      break;
    }
    else{
      stop();
      delay(1000);
    }
  }

  if (start_succes){
    Serial.println(F("success starting bno"));
  }
  else{
    Serial.println(F("no success starting bno"));
    stop();

    Serial.println(F("we reboot"));
    while(true){;}
  }
  
  return start_succes;
}

[adamgarbo]

Hi folks,

I tested the majority of BNO080 examples using v2.0.5 of the Apollo3 Core and a SparkFun Redboard Artemis/Redboard Artemis Nano. I would have also included theMicroMod Artemis, but v2.0.5 doesn't have this board definition yet.

I am happy to report that I was not able to reproduce any of the I2C hangs or sensor initialization failures with the examples I tested. Perhaps this bug is specific only to v1.2.x?

@jerabaul29, with SparkFun announcing the EOL of v1.x of the Apollo3 Core yesterday, I believe it makes sense for us to focus our efforts on v2.x. User contributions will be a big help in getting it to where it needs to be.

Edit: I may have spoken too soon. I left Example 17 running and it crashed after ~20 minutes multiple times. I was then unable to re-initialize the sensor until I reset/unplugged it. Yikes. I wonder if the Hillcrest Labs/CEVA would have any insight.

Cheers,
Adam