Token file not found "token"

[harshakhmk]

I am using fastembed-rs package, to create embeddings for text input
While I am running my application as a shared library on Android, it is giving below error

hf_hub: Token file not found "token"
ureq:::stream connecting to huggingface.co 

use hf_hub::api::sync::Api;

let api = Api::new().unwrap();

let repo = api.model("bert-base-uncased".to_string());
let _filename = repo.get("config.json").unwrap();

// filename  is now the local location within hf cache of the config.json file

[harshakhmk]

Using the above code in my sample rust crate which is built as a shared library and used in Android app, app is crashing

[harshakhmk]

@Narsil any leads on this would be highly helpful

[polarathene]

Here is the rough flow of what's happening when you call Api::new():

Click to view

hf-hub/src/api/sync.rs

Lines 258 to 262 in 9d6502f

	impl Api {
	/// Creates a default Api, for Api options See [`ApiBuilder`]
	pub fn new() -> Result<Self, ApiError> {
	ApiBuilder::new().build()
	}

hf-hub/src/api/sync.rs

Lines 99 to 108 in 9d6502f

	impl ApiBuilder {
	/// Default api builder
	/// ```
	/// use hf_hub::api::sync::ApiBuilder;
	/// let api = ApiBuilder::new().build().unwrap();
	/// ```
	pub fn new() -> Self {
	let cache = Cache::default();
	Self::from_cache(cache)
	}

hf-hub/src/lib.rs

Lines 194 to 208 in 9d6502f

	impl Default for Cache {
	fn default() -> Self {
	let mut path = match std::env::var("HF_HOME") {
	Ok(home) => home.into(),
	Err(_) => {
	let mut cache = dirs::home_dir().expect("Cache directory cannot be found");
	cache.push(".cache");
	cache.push("huggingface");
	cache
	}
	};
	path.push("hub");
	Self::new(path)
	}
	}

hf-hub/src/api/sync.rs

Lines 117 to 129 in 9d6502f

	pub fn from_cache(cache: Cache) -> Self {
	let token = cache.token();
	let progress = true;
	Self {
	endpoint: "https://huggingface.co".to_string(),
	url_template: "{endpoint}/{repo_id}/resolve/{revision}/{filename}".to_string(),
	cache,
	token,
	progress,
	}
	}

hf-hub/src/api/sync.rs

Lines 160 to 182 in 9d6502f

	pub fn build(self) -> Result<Api, ApiError> {
	let headers = self.build_headers();
	let agent = ureq::builder().try_proxy_from_env(true).build();
	let client = HeaderAgent::new(agent, headers.clone());
	let no_redirect_agent = ureq::builder()
	.try_proxy_from_env(true)
	.redirects(0)
	.build();
	let no_redirect_client = HeaderAgent::new(no_redirect_agent, headers);
	Ok(Api {
	endpoint: self.endpoint,
	url_template: self.url_template,
	cache: self.cache,
	client,
	no_redirect_client,
	progress: self.progress,
	})
	}
	}

For the cache.token() call in from_cache():

hf-hub/src/lib.rs

Lines 41 to 68 in 9d6502f

	/// Returns the location of the token file
	pub fn token_path(&self) -> PathBuf {
	let mut path = self.path.clone();
	// Remove `"hub"`
	path.pop();
	path.push("token");
	path
	}
	/// Returns the token value if it exists in the cache
	/// Use `huggingface-cli login` to set it up.
	pub fn token(&self) -> Option<String> {
	let token_filename = self.token_path();
	if !token_filename.exists() {
	log::info!("Token file not found {token_filename:?}");
	}
	match std::fs::read_to_string(token_filename) {
	Ok(token_content) => {
	let token_content = token_content.trim();
	if token_content.is_empty() {
	None
	} else {
	Some(token_content.to_string())
	}
	}
	Err(_) => None,
	}
	}

---

So what's happening is:

1. A cache directory for HF to use is checked via the ENV HF_HOME, otherwise it defaults to ~/.cache/huggingface/hub for the cache directory.
2. Then when the API struct is created, it takes this path and checks the parent dir (omitting hub) to look for a file named token, thus default path is ~/.cache/huggingface/token.
3. That token file check is part of calling from_cache() internally to create the API builder struct, which itself calls cache.token() on that path value from step 1. When this does not exist you get the info log you encountered.
4. That presently cannot be avoided easily, you'd have to manually add a lot more code to create an API builder without the cache.token() call.

That shouldn't be causing a crash though.

[i-am-logger]

i'll add to this
i get the following

[2024-11-12T01:39:38Z INFO  hf_hub] Token file not found "/home/snick/.cache/huggingface/token"

two issues here

1. it ignored the cache dir (used with_cache_dir) , it used a default location
2. it should be a WARN not INFO ;)