libraries: "SoftwareSerial" does not work.

[RobTillaart]

Repo: https://github.com/RobTillaart/MHZCO2

Has an example that uses SoftwareSerial and it compiles well on the desktop.
However in the build they fail to build

I have added the following in the yml file but it still

  libraries:
    - "SoftwareSerial"

Note that software serial is one of the "built-in" libraries

So for now I just commented the SoftwareSerial line, so that the other things are build/tested.

---

error output - https://github.com/RobTillaart/MHZCO2/actions/runs/3950260138/jobs/6762557012

Discovered example sketch...                                  MHZCO2_serial1.ino
Using configuration override from example... /home/runner/work/MHZCO2/MHZCO2/examples/MHZCO2_serial1/.arduino-ci.yml
Wildcard architecture in library.properties, using configured platforms... 
    mega2560
...Wildcard architecture in library.properties, using configured platforms      
Board package arduino:avr has a defined URL...                                 ✓
Installing board package arduino:avr...                                        ✓
Installing dependency of <compile/libraries>: 'SoftwareSerial'...              ✗
This may indicate a problem with your configuration; halting here
Failures: 1
========== Last backend command (if relevant):
 $  /home/runner/arduino-cli --format json lib install --no-deps SoftwareSerial
========== Backend Stdout:

========== Backend Stderr:
Error installing SoftwareSerial: Library 'SoftwareSerial' not found

[ianfixes]

This seems related to a problem I had earlier with Ethernet: #192 (comment)

I can't recall how I fixed that one, but this comment is my "note to self".

[ianfixes]

The main question here is what the official project will try to do if you list SoftwareSerial as a dependency in library.properties. Is that just a no-op? @per1234 I'm hoping you can add more insight than what I would get if I try to just reverse engineer the behavior.

[per1234]

"SoftwareSerial" is a platform bundled library. It is distributed as a component of some (but not all) boards platforms such as arduino:avr:

https://github.com/arduino/ArduinoCore-avr/tree/master/libraries/SoftwareSerial

The depends field of library.properties can only be used to list libraries that are in the Arduino Library Manager index. The "SoftwareSerial" library is not in the Arduino Library Manager index, so it does not make sense to add it to the depends field. The closest thing is the architectures field, where you could specify compatibility only with the architectures (e.g., architectures=avr,megaavr) of the specific boards platforms that provide a "SoftwareSerial" library.

[RobTillaart]

In practice that means I have to disable the compile tests for examples using SWS for those boards.
It might be worth adding a few lines on this in the section about libraries

Thanks @per1234 for this insight

[ianfixes]

Thanks @per1234, that's extremely helpful information here. I was thinking about how my library should approach this, because if I were to maintain my own mapping of "platform bundled libraries" in this project, I am in danger of being perpetually out of sync with the "ground truth" known by the Arduino environment itself. Still, is there any information I might be able to use from the configuration (e.g. default.yml) to check the local filesystem for a platform? Is that a wildly unstable idea?

@RobTillaart, you might also be able to just provide an example-specific override using an .arduino-ci.yml file in that example's directory, that omits the SoftwareSerial dependency.

[RobTillaart]

@ianfixes
used the following .arduino-ci.yml to compile the example for the platforms that support SoftwareSerial

See https://github.com/RobTillaart/MHZCO2/tree/master/examples/MHZCO2_sw_serial

platforms:
  rpipico:
    board: rp2040:rp2040:rpipico
    package: rp2040:rp2040
    gcc:
      features:
      defines:
        - ARDUINO_ARCH_RP2040
      warnings:
      flags:

packages:
  rp2040:rp2040:
    url: https://github.com/earlephilhower/arduino-pico/releases/download/global/package_rp2040_index.json

compile:
  #  Choosing to run compilation tests on 2 different Arduino platforms
  platforms:
    - uno
    # - due      //  has no SoftwareSerial
    # - zero     //  has no SoftwareSerial
    - leonardo
    # - m4       //  has no SoftwareSerial
    # - esp32    //  has no SoftwareSerial
    - esp8266
    - mega2560
    - rpipico
  #  external libraries
  libraries:
    # - "SoftwareSerial" does not work as it is build in.

[ianfixes]

I'm still not grasping what the proper configuration might be in this case. On one hand, per1234 says that SoftwareSerial is not in the Arduino Library Manager index -- this means specifying it in .arduino-ci.yml will always fail. So perhaps it means that the library is not compatible with those other platforms?

If that's the case, then it would imply that this is a problem of configuration and not something that arduino_ci itself can help with.

If there isn't a straightforward test that I can do from the library to determine whether a library is possibly builtin (like SoftwareSerial), then I'm not sure what to do here because I can't imagine how I'd improve the messaging that's already being returned -- the library just doesn't exist in the library manager.

[RobTillaart]

SWS is build in so the platforms that have it will compile. (we also not include "Arduino Lib" )
Mention SWS as external lib fails as it is no external lib.
A quick look in the AVR SWS code shows some non-portable parts.

determine whether a library is possibly builtin

Now if I add an "external library" that does not exist as ext. lib in the yml file, I get an error. full stop.
It might help to extend the error message with
"do not mention build in libraries here, exclude platforms that do not support them"

Is not the solution, but will help users a lot.

[per1234]

is there any information I might be able to use from the configuration (e.g. default.yml) to check the local filesystem for a platform?

You can get the list of installed platforms using arduino-cli core list (using a flag like --format json to get machine readable output from the command).

The platform will be installed at this location:

<directories.data>/packages/<vendor>/hardware/<architecture>

• You can get <directories.data> from arduino-cli config dump
• <vendor> is the first component of the platform ID shown in the arduino-cli core list output (e.g., the arduino from arduino:avr).
• <architecture> is the second component of the platform ID shown in the arduino-cli core list output (e.g., the avr from arduino:avr).

The platform bundled libraries are installed in the libraries subfolder of the platform folder.

You can also get the location of the libraries directly using arduino lib list --all --format json --fqbn <fqbn of target board> [library name]. The library.location field indicates whether it is a "built-in" (value 0), user (value 1), platform bundled (value 2), or reference platform bundled (value 3) library (reference).

<directories.data>/packages/<vendor>/hardware/<avr>

perhaps it means that the library is not compatible with those other platforms?

Yes. "Platform bundled libraries" are typically architecture-specific fundamental libraries.

A better example of a platform bundled library is the "Wire" library. Most boards platform have a bundled "Wire" library. Each platform's Wire library uses a compatible standardized API, but the implementation code can be very different from one platform to the next due to it being architecture specific. For this reason, it would not make sense to try to maintain a single global "Wire" library that is compatible with all of the ever increasing number of architectures people have created boards platforms for.

"SoftwareSerial" didn't attain the same level of universal adoption by the boards platform developers as "Wire" and "SPI". Some of the newer target microcontrollers have multiple hardware serial ports, in some cases even multiplexed (example), so the developers might have felt it was not worth the effort to write a variant of the library for their architecture.

There is another wrinkle here, which is that some platforms bundle a variant of a library that is available from the Arduino Library Manager. They typically do this in cases where the code of the commonly used library is not compatible with their architecture. For example, Arduino maintains a popular library named "SD". The esp8266 boards platform has a platform bundled SD library written for use on the ESP8266 microcontroller:

https://github.com/esp8266/Arduino/tree/master/libraries/SD

do not mention build in libraries here

"Built-in" is the wrong term. The correct term for libraries like "SoftwareSerial" is "platform bundled":

https://arduino.github.io/arduino-cli/latest/platform-specification/#platform-bundled-libraries

The "built-in" term is used to refer to a separate class of libraries that are included as part of the Arduino IDE installation, such as "Servo", "Mouse", "Ethernet". Although they are installed by default with Arduino IDE, the "built-in" libraries are also in the Library Manager index, so you can use Library Manager to install the "built-in" libraries. In fact, the Library Manager infrastructure is even used by Arduino IDE 2.x to install the "built-in" libraries on the first run after a new installation.

[probonopd]

Also running into this.

https://github.com/probonopd/TAPController/actions/runs/5705624580/job/15460490942#step:3:155

Is there a solution?

[ianfixes]

It's been a while since I looked at this. I can see that there are a few things I need to clean up terminology-wise ("built-in" -> "platform-bundled"), and then I think there is some work to figure out exactly what is and isn't platform-bundled and react to it somehow.

In truth, I'm currently one of the least knowledgeable people in this thread about how everything works. I lean on the experts for how to make decisions at config/compile time, and work from that. There may be a workaround in the configuration for now by messing with the compiler flags on a platform to force HAVE_HWSERIAL0?