Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modeled Coverage - Phase 3 #559

Closed
wants to merge 51 commits into from
Closed
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
0479b5f
Add ingest service for coverage object
maplant Jun 21, 2023
498685e
implement basic algorithm for computing coverage points
maplant Jun 26, 2023
bab457f
Fix bug where multiple signal levels would get mixed
maplant Jun 26, 2023
60814f7
Implement logic for modeled coverage
maplant Jun 27, 2023
e52dc60
Fmt
maplant Jun 27, 2023
e35ee73
Convert rewarding to take coverage into account
maplant Jun 28, 2023
7e04c77
Convert over most of the reward share tests, add fixes
Jul 3, 2023
54bcaed
Port over one more test
Jul 5, 2023
ca0434d
Address remaining issues
Jul 7, 2023
da1ccb2
Update file_store/src/error.rs
Jul 7, 2023
8a1e2a6
Update mobile_verifier/src/coverage.rs
Jul 10, 2023
57da60b
Fix tests, add setting, improve some of the code
maplant Jul 10, 2023
ba97a7e
Add coverage_object column to heartbeats
maplant Jul 10, 2023
74d6eb8
s/no/none/g
maplant Jul 10, 2023
64fa0b3
Correct hex_coverage table
maplant Jul 11, 2023
8c52ceb
Merge remote-tracking branch 'origin/main' into map/modeled-coverage
maplant Jul 11, 2023
a1c254d
Fmt and Fix
maplant Jul 11, 2023
6619bfd
Move coverage object around to satisfy OCD
maplant Jul 11, 2023
99505a3
Clippy
maplant Jul 12, 2023
6b5c69e
Fix CellHeartbeat to allow backwards compat w/ empty coverage_object
maplant Jul 12, 2023
035e4a9
Fix coverage claim time with first_timestamp
maplant Jul 12, 2023
242ad5a
Address comments
maplant Jul 13, 2023
349a92e
Move coverage claim time adjustment outside of stream
maplant Jul 13, 2023
567bd35
Move coverage claim time adjustment into heartbeat save
maplant Jul 13, 2023
a5c9d4d
Adjust migration
maplant Jul 17, 2023
9d1dffb
Move things around and cache
maplant Jul 17, 2023
0f5cd3d
Add fun todo :-)
maplant Jul 17, 2023
cb0ae3c
Correct seniority updates
maplant Jul 18, 2023
d886af8
Merge remote-tracking branch 'origin/main' into map/modeled-coverage
maplant Jul 18, 2023
5cd1b75
Fix coverage claim time adjustment
maplant Jul 18, 2023
fee3eb3
Fix seniority
maplant Jul 19, 2023
e35c82b
CMove to insert or update model for seniority
maplant Jul 19, 2023
985c990
Refactor coverage and heartbeats a little bit
maplant Jul 21, 2023
d066c29
Merge remote-tracking branch 'origin/main' into map/modeled-coverage
maplant Jul 24, 2023
a71a601
Refactor into_rewards a bit
maplant Jul 24, 2023
1deea42
Clippy
maplant Jul 24, 2023
9c88a27
Delete old hex coverages
maplant Jul 24, 2023
45638d5
Fix deletion and validated heartbeats stream
maplant Jul 25, 2023
8201e1b
Unnet into_rewards
maplant Jul 26, 2023
9301ad7
Clippy
maplant Jul 27, 2023
b8b719b
Embed heartbeat object
maplant Aug 2, 2023
eccb3f5
Add cbsd_id to seniority deletion
maplant Aug 16, 2023
c2827a8
Address issues with seniority
maplant Aug 16, 2023
83f0ce6
Merge remote-tracking branch 'origin/main' into map/modeled-coverage
maplant Aug 22, 2023
12b9c96
Use timestamp instead of datetime
maplant Aug 22, 2023
e288a10
Refactor seniority updates to make them more easily testable
maplant Aug 22, 2023
cf76f30
First seniority unit test
maplant Aug 22, 2023
25d787a
Add tests for seniority updates
maplant Aug 22, 2023
34057e7
Merge remote-tracking branch 'origin/main' into map/modeled-coverage
maplant Aug 22, 2023
9ec0f69
Add initial coverage object tests
maplant Aug 22, 2023
9b38ab4
Add modeled coverage integration tests
maplant Aug 23, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 50 additions & 15 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 4 additions & 3 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ sqlx = {version = "0", features = [
]}

helium-crypto = {version = "0.6.8", features=["sqlx-postgres", "multisig"]}
helium-proto = {git = "https://github.com/helium/proto", branch = "master", features = ["services"]}
helium-proto = {git = "https://github.com/helium/proto", branch = "map/modeled-coverage", features = ["services"]}
maplant marked this conversation as resolved.
Show resolved Hide resolved
hextree = "*"
solana-client = "1.14"
solana-sdk = "1.14"
Expand All @@ -83,8 +83,8 @@ prost = "*"
once_cell = "1"
lazy_static = "1"
config = {version="0", default-features=false, features=["toml"]}
h3o = "0"
xorf = {version = "0", features = ["serde"] }
h3o = {version = "0", features = ["serde"]}
xorf = {version = "0", features = ["serde"]}
bytes = "*"
structopt = "0"
bincode = "1"
Expand All @@ -97,6 +97,7 @@ itertools = "*"
data-credits = {git = "https://github.com/helium/helium-program-library.git", tag = "v0.1.0"}
helium-sub-daos = {git = "https://github.com/helium/helium-program-library.git", tag = "v0.1.0"}
price-oracle = {git = "https://github.com/helium/helium-program-library.git", tag = "v0.1.0"}
uuid = {version = "1.3.4", features = ["v4", "serde"]}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe not a concern depending on the expected volume of coverage objects, but ulid-rs may be good to keep in mind for "lexicographically sortable" IDs that are UUIDv4 compatible to prevent too much fragmentation in the database

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would absolutely prefer ulids, but we are given the uuids. we don't decide them


[patch.crates-io]
sqlx = { git = "https://github.com/helium/sqlx.git", rev = "92a2268f02e0cac6fccb34d3e926347071dbb88d" }
2 changes: 2 additions & 0 deletions file_store/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ sqlx = {workspace = true}
async-trait = {workspace = true}
derive_builder = "0"
retainer = {workspace = true}
uuid = {workspace = true}
h3o = {workspace = true}

[dev-dependencies]
hex-literal = "0"
Expand Down
102 changes: 102 additions & 0 deletions file_store/src/coverage.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
use crate::{
error::DecodeError,
traits::{MsgDecode, TimestampDecode},
Error, Result,
};
use chrono::{DateTime, Utc};
use helium_crypto::PublicKeyBinary;
use helium_proto::services::poc_mobile::{
CoverageObjectIngestReportV1, CoverageObjectReqV1,
RadioHexSignalLevel as RadioHexSignalLevelProto, SignalLevel,
};
use serde::{Deserialize, Serialize};
use uuid::Uuid;

#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct RadioHexSignalLevel {
pub location: h3o::CellIndex,
pub signal_level: SignalLevel,
pub signal_power: f64,
}

#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct CoverageObject {
pub pub_key: PublicKeyBinary,
pub uuid: Uuid,
pub cbsd_id: String,
pub coverage_claim_time: DateTime<Utc>,
pub indoor: bool,
pub coverage: Vec<RadioHexSignalLevel>,
}

#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct CoverageObjectIngestReport {
pub received_timestamp: DateTime<Utc>,
pub report: CoverageObject,
}

impl MsgDecode for CoverageObject {
type Msg = CoverageObjectReqV1;
}

impl MsgDecode for CoverageObjectIngestReport {
type Msg = CoverageObjectIngestReportV1;
}

impl TryFrom<CoverageObjectIngestReportV1> for CoverageObjectIngestReport {
type Error = Error;

fn try_from(v: CoverageObjectIngestReportV1) -> Result<Self> {
Ok(Self {
received_timestamp: v.received_timestamp.to_timestamp_millis()?,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason not to implement the MsgTimestamp trait to handle these timestamps here and in subsequent msgs below ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh, just didn't see a reason to implement it to only be used one other place

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think its a matter of convention and ensuring consistency.

report: v
.report
.ok_or_else(|| Error::not_found("ingest coverage object report"))?
.try_into()?,
})
}
}

impl TryFrom<CoverageObjectReqV1> for CoverageObject {
type Error = Error;

fn try_from(v: CoverageObjectReqV1) -> Result<Self> {
let coverage: Result<Vec<RadioHexSignalLevel>> = v
.coverage
.into_iter()
.map(RadioHexSignalLevel::try_from)
.collect();
Ok(Self {
pub_key: v.pub_key.into(),
uuid: Uuid::from_slice(&v.uuid).map_err(DecodeError::from)?,
cbsd_id: v.cbsd_id,
coverage_claim_time: v.coverage_claim_time.to_timestamp()?,
indoor: v.indoor,
coverage: coverage?,
})
}
}

impl TryFrom<RadioHexSignalLevelProto> for RadioHexSignalLevel {
type Error = Error;

fn try_from(v: RadioHexSignalLevelProto) -> Result<Self> {
Ok(Self {
signal_level: SignalLevel::from_i32(v.signal_level).ok_or_else(|| {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you use v.signal_level() or are you explicitly avoiding the proto behavior of using the default (lowest/first) element of the signal level enum if the oracle doing the try_from() has an older version of the proto code that doesn't account for the variant in a newer message?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This just happens to be the pattern I see everywhere else in the file store

DecodeError::unsupported_signal_level("coverage_object_req_v1", v.signal_level)
})?,
signal_power: v.signal_power,
location: v.location.parse().map_err(DecodeError::from)?,
})
}
}

impl Into<RadioHexSignalLevelProto> for RadioHexSignalLevel {
maplant marked this conversation as resolved.
Show resolved Hide resolved
fn into(self) -> RadioHexSignalLevelProto {
RadioHexSignalLevelProto {
signal_level: self.signal_level as i32,
signal_power: self.signal_power,
location: self.location.to_string(),
}
}
}
10 changes: 10 additions & 0 deletions file_store/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,14 @@ pub enum DecodeError {
UnsupportedParticipantSide(String, i32),
#[error("unsupported verification status, type: {0}, value: {1}")]
UnsupportedStatusReason(String, i32),
#[error("unsupported signal level, type: {0}, value: {1}")]
UnsupportedSignalLevel(String, i32),
#[error("invalid unix timestamp {0}")]
InvalidTimestamp(u64),
#[error("Uuid error: {0}")]
UuidError(#[from] uuid::Error),
#[error("Ivalid cell index error: {0}")]
maplant marked this conversation as resolved.
Show resolved Hide resolved
InvalidCellIndexError(#[from] h3o::error::InvalidCellIndex),
}

#[derive(Error, Debug)]
Expand Down Expand Up @@ -127,6 +133,10 @@ impl DecodeError {
pub fn unsupported_status_reason<E: ToString>(msg1: E, msg2: i32) -> Error {
Error::Decode(Self::UnsupportedInvalidReason(msg1.to_string(), msg2))
}

pub fn unsupported_signal_level(msg1: impl ToString, msg2: i32) -> Error {
Error::Decode(Self::UnsupportedSignalLevel(msg1.to_string(), msg2))
}
}

impl From<helium_crypto::Error> for Error {
Expand Down
Loading