Slack Archive

To Index

Back To Top
<@U024KC347B4> has joined the channel
<@U025GFV1SJC> has joined the channel
This content can't be displayed.
/github subscribe weboftrust/cesride
:white_check_mark: Subscribed to . This channel will receive notifications for `issues`, `pulls`, `commits`, `releases`, `deployments`
/github subscribe weboftrust/parside
:white_check_mark: Subscribed to . This channel will receive notifications for `issues`, `pulls`, `commits`, `releases`, `deployments`
<@U04HQD29Z7E> has joined the channel
Over the weekend I implemented `Matter` in `cesride` at parity with `KERIpy` (I believe). Here's the code: It can probably use some more work, I'm a fairly new Rust coder (2 months but pretty intensive) but it's a good start. <@U024KC347B4> you probably want to check this out as I noticed you made some changes, too. I can open a PR for review if people want? I'm willing to contribute a great deal to this suite of projects in Rust. I tested this by instantiating `Matter` several different ways, in one gross function. It grew out of what was there, I just kind of took the path of least resistance but this is where my work could benefit the most, I believe. Better tests. I will add them later. I did generate lots of tests for `Sizages` and `Codes` though. Normally I wouldn't throw down a 1300 line PR but this is a foundational bit that was hard to get away from. For the curious, I used string transformations in Ruby to turn Python codexes etc into Rust code directly.
Hey Jason, yes I was working on Matter so we'll likely conflict
Now that I know of this channel, after we resolve the conflict can I just grab issues?
Or should we have some sort of regular grooming?
<@U04BRS1MUAH> has joined the channel
Just realized I left a couple magic numbers in that branch too, around the variable length binary decoding
yeah, open a PR if you could rebase locally before and make it a single commit it will help with house keeping and a cleaner git history over time
As for process, we have a cesr call every other Thursday and a keri call (today) other than that assigning yourself an issue is about as formal as it gets
Okay cool, thanks Kevin! I'll open a PR shortly
Let’s get your branch merged in today, I had other cosmetic changes in there, moving size tests into sizeage and hard tables and tests into their own file along with matter_codex
Just to try reduce the size or core::matter
Yeah those are definitely good changes to make I was just hacking it out for functionality. I am all for smaller/focused files
Let me know if you need any changes made to my PR, and how to proceed
<@U04HMQT1XFV> has joined the channel
joseph.l.hunsaker
<@U03QRSUA87Q> has joined the channel
<@U024CJMG22J> has joined the channel
<@U03EUG009MY> has joined the channel
<@U02PA6UQ6BV> has joined the channel
You are on fire Jason! Keep it up.
:pray:
<@U02MD0HA7EJ> has joined the channel
rodolfo.miranda
<@U03P53FCYB1> has joined the channel
:tada:
Screenshot 2023-01-27 at 5.57.42 PM.png
<@U048YRLFKTK> has joined the channel
I have code that passes this test module:
mod test {
    use super::{data, Data, Value};

    #[test]
    fn data_macro() {
        let x: i64 = -1234567890;
        let s = "string".to_string();

        let mut d = data!({
            "thing": 2,
            "other thing": [&s, 1.666, x, true, {"nested array": [{}, []]}],
            "last thing": null
        });

        // to_json()
        assert_eq!(
            d.to_json().unwrap(),
            "{\"thing\":2,\"other thing\":[\"string\",1.666,-1234567890,true,{\"nested array\":[{},[]]}],\"last thing\":null}"
        );

        // query/indexing
        assert_eq!(d["thing"], d[0]);  // we can index into an object with an integer or string key
        assert_ne!(d["thing"], d[1]);

        // display
        assert_eq!(format!("{}", d["thing"]), "2");

        // data extraction
        assert_eq!(d["last thing"], Value::Null);
        assert_eq!(d["other thing"][3].to_bool(), true);
        assert_eq!(d["thing"].to_i64(), 2);
        assert_eq!(d["other thing"][1].to_f64(), 1.666);
        assert_eq!(d["other thing"][0].to_string(), "string");
        assert_eq!(d["other thing"][4]["nested array"][1].to_vec(), vec![]);
        assert_eq!(d["other thing"][4]["nested array"][0].to_map(), indexmap::IndexMap::new());

        // mutability
        d["thing"] = data!({"something more complex": {"key": 987654321 }});
        assert_eq!(
            d.to_json().unwrap(),
            "{\"thing\":{\"something more complex\":{\"key\":987654321}},\"other thing\":[\"string\",1.666,-1234567890,true,{\"nested array\":[{},[]]}],\"last thing\":null}"
        );

        // serde_json parsing interop
        let v: serde_json::Value = serde_json::from_str(&d.to_json().unwrap()).unwrap();
        let d2 = Value::from(&v);
        assert_eq!(d.to_json().unwrap(), d2.to_json().unwrap());
    }
}
I was thinking it would be useful to have such a structure in Rust for things like KEDs and SADs and now that I've built the core of it, once `cesride` is complete it should be trivial to implement `.to_cesr()` and `.to_cesrb()`. I was thinking since this complements parsing logic, all this probably wants to live in `parside`. Any objections to me starting work on spiking out `parside` and spiking out an integration with `cesride`?
The hard part was building the macros, and I stole most of that code from `serde_json` so it should be pretty solid.
Also it may not be clear from the example but you don't need to use a real `String` to create data, you can pass a `&str`.
data!({"foo": "bar"})
is valid.
This is actually really cool to do in Rust, things like this aren't common in compiled languages. I wanted to make the underlying data access simpler and tried for a while using things like `Any` but was unsuccessful. Since `serde_json` does almost exactly the same thing with respect to its `Value` objects I figure it's probably impossible to do elegantly.
My theory is that most classes in `KERIpy` that belong in `parside` can be built from stuff like this, and if we find we need the `data!()` macro in `cesride` it may be a sign we should move something up to `parside`.
But I haven't read all the code.
<@U04HQD29Z7E> could you please push the code somewhere to let me have a look? I'm worrying about does it friendly for FFI.
This code wouldn’t need to pass through an ffi
Serialized data would enter parside which can call rust directly
The ffi of cesride can be addressed separately, sorry if I misspoke
I am writing up a simple FFI for parside right now to show you what it looks like, and I'll generate the C header file to give you an idea of what integration is like
Cool. BTW my plan is to use UniFFI for Kotlin/Python bindings and wasm for Javascript. I plan to start work on parside and FFI from Monday so let's be in sync on that to not waste time.
Sure, there are issues we can use to coordinate.
mod core;
mod data;
mod error;

pub struct Parser;

macro_rules! boxed {
    ($e:expr) => { Box::into_raw(Box::new($e)) }
}

#[no_mangle]
pub unsafe extern "C" fn parside_parser_new() -> *mut Parser {
    boxed!(Parser {})
}

#[no_mangle]
pub unsafe extern "C" fn parside_inhale(
    ser: *const libc::c_uchar,
    ser_sz: libc::size_t,
    parser: *const Parser,
) -> libc::c_int {
    if parser.is_null() || ser.is_null() {
        return 1;
    }

    // convert ser into octet string
    let ser = unsafe {
        std::slice::from_raw_parts(ser, ser_sz)
    };

    // grab parser from opaque pointer
    let mut parser = unsafe {
        &*parser
    };
    
    // parse octet string
    parser.inhale(ser)?;

    return 0;
}

#[no_mangle]
pub unsafe extern "C" fn parside_next(
    parser: *const Parser
) -> *mut Value {
    let parser = unsafe {
        assert!(!parser.is_null());
        &*parser
    };

    boxed!(parser.next())
}

#[no_mangle]
pub unsafe extern "C" fn parside_free(
    object: *mut libc::c_void
) {
    if !object.is_null() { unsafe {
        Box::from_raw(object);
    }}
}
this is a start, i wrote it up in a few minutes so it won't compile but it shows you the idea. generally i like returning an int to indicate success or a failure code and dealing with all returned data as inout params, but that's not exactly the example I am showing here
I got the ffi to work
❯ cat header.h
#include <stdarg.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdlib.h>

typedef struct Parser Parser;

void parser_free(struct Parser *parser);

struct Parser *parser_new(void);

int parser_inhale(const unsigned char *ser, size_t ser_sz, struct Parser *parser);

int parser_next(const struct Parser *parser);
❯ cat main.c 
#include "header.h"

int main(int argc, char *argv[]) {
    unsigned char s[] = "test string";

    Parser* parser = parser_new();
    parser_inhale(s, 11, parser);
    parser_next(parser);
    parser_free(parser);
    
    return 0;
}
❯ clang main.c -o main -L target/release -l saidide
❯ ./main 
test string
This is the rust code:
pub struct Parser {
    data: String
}

impl Parser {
    pub fn inhale(&mut self, ser: &[u8]) {
        self.data = std::str::from_utf8(ser).unwrap().to_string();
    }

    pub fn dump(&self) {
        println!("{}", self.data);
    }
}

#[no_mangle]
pub extern "C" fn parser_free(parser: *mut Parser) {
    unsafe { drop(opaque_pointer::own_back(parser)) };
}

#[no_mangle]
pub extern "C" fn parser_new() -> *mut Parser {
    opaque_pointer::raw(Parser { data: "data".to_string() })
}


#[no_mangle]
pub unsafe extern "C" fn parser_inhale(
    ser: *const libc::c_uchar,
    ser_sz: libc::size_t,
    parser: *mut Parser,
) -> libc::c_int {
    if parser.is_null() || ser.is_null() {
        return 1;
    }

    // convert ser into octet string
    let ser = unsafe {
        std::slice::from_raw_parts(ser, ser_sz)
    };

    // grab parser from opaque pointer
    let parser = unsafe {
        match opaque_pointer::mut_object(parser) {
            Ok(p) => p,
            Err(_) => { return 1; },
        }
    };
    
    // parse octet string
    parser.inhale(ser);

    return 0;
}

#[no_mangle]
pub unsafe extern "C" fn parser_next(
    parser: *const Parser
) -> libc::c_int {
    if parser.is_null() {
        return 1;
    }

    let parser = unsafe {
        match opaque_pointer::object(parser) {
            Ok(p) => p,
            Err(_) => { return 1; },
        }
    };

    parser.dump();
    return 0;
}
it obviously doesn't do anything but it shows how we can use a struct in rust through the FFI
i mean, it prints a string that we store. it does _something_. but not much
the header is generated with cbindgen, the lib is setup to build a staticlib. I'll add all this to `parside` this weekend to give you a good head start
This looks pretty nice at a glance, for the python use case:
You can create multiple Cargo.toml files and have each point to a different that depends on common code, and expose multiple FFIs. This is a good thing for us.
This sounds very cool
Yes I was wrestling with how to expose this to multiple languages and take advantage of Rust's nice integrations
Love to put this on the agenda for next meeting
wow, this is amazing to see coming together. I hope to contribute soon
I threw together a draft PR to spike things out a bit in `cesride`: Would love some :eyes:
I dropped a simple PR that shows how a python integration may work. Obviously not for everyone and a straight C interface will be most widely usable, but it's a neat idea.
❯ make python-shell
Python 3.10.6 (main, Aug 11 2022, 13:49:25) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from cesride import Matter
>>> m = Matter(qb64="BGlOiUdp5sMmfotHfCWQKEzWR91C72AH0lT84c0um-Qj")
>>> qb2 = m.qb2()
>>> print(qb2)
[4, 105, 78, 137, 71, 105, 230, 195, 38, 126, 139, 71, 124, 37, 144, 40, 76, 214, 71, 221, 66, 239, 96, 7, 210, 84, 252, 225, 205, 46, 155, 228, 35]
>>> m2 = Matter(qb2=bytes(qb2))
>>> m2.qb64()
'BGlOiUdp5sMmfotHfCWQKEzWR91C72AH0lT84c0um-Qj'
>>> 
This is so exciting to see all of this activity happening! I can’t wait to dive into the code.
If anyone is interested, here's a POC I did earlier in the week on arbitrary value objects for use in `parside`: If it makes sense we can pull it in.
<@U048YRLFKTK> I didn't realize when you mentioned `uniffi` that it was actually a rust package that did this! Let us know if you make progress!
artem.ivanov.konst
<@U04EFUFR71Q> has joined the channel
^ I posted a comment on the above Issue. I recommend, since slack is not permanent at our tier, that we discuss issues like this in the comments on GitHub for posterity.
What should our criterion for consensus be on issues like this?
artem.ivanov.konst
Hey, Why do we have `Matter` as `structure` and `Verfer`, `Diger`, etc as `traits`? What’s idea behind this? As for me it should be opposite. if we want to define different kinds of primitives there should be defined `Matter` as `trait` and add structures `Verfer` , `Diger` , etc which implements `Matter`, Groups will consists of various primitives. With currently implemented approach, if I have `Matter` instance I can call any methods from all implemented traits `Verfer` `Diger` etc . That seems to be wrong. Using `trait` as a fields of `Group` structure adds a lot of complexity because we have to wrap it into `Box` .
It was the idea behind moving quickly and people learning Rust as they went, with a focus on porting as much logic as we could while we had resources available to bootstrap a project that had been idle for six months. Glad you're here now though. Correct Matter is essentially an abstract class not something you would initialize directly, the others are.
Yeah it was to move fast. But I have actually thought about this too Artem, that it should likely be all structs with `matter` hidden under the hood.
As <@U04EFUFR71Q> pointed out the current implementation of Matter the subsequent derived classes isn't ideal. I created and to contain the discussion. I think 72 is a good first refactor that will be less painful than 73, but allow consumers of cesride to progress. Then 73 can be discussed and addressed. Looking for feedback <@U04EFUFR71Q> <@U04HQD29Z7E>
I am going to start looking at 72.
for visibility, working on the Matter refactor:
artem.ivanov.konst
<@U04HQD29Z7E> Could you take a look this [comment]() regarding parside I added yesterday.
artem.ivanov.konst
Right now, I do not see way how to implement option with `generic` parsing result after the latest cesride changes
I will look this morning.
I already did, a bit
I left a comment, I may have thought of a solution.
I am not sure, being a new Rustacean.
Also that link has an extra k in `parside` and is broken
wip-ing
contact.johannes
<@U04PXR871PS> has joined the channel
contact.johannes
<@U04PXR871PS> has joined the channel
<@U04HQD29Z7E> <@U024KC347B4> a cleanup of clippy warnings for cesride ^^^
I don't really know how we feel about a release candidate in the library proper, but this PR can stay open until it lands if we don't want to merge it. The changes are positive.
I think it is a discussion for when/if we decide to cut a 1.0.0 I think we have a few options.
<@U055XBX2EAD> has joined the channel
<@U056E1W01K4> has joined the channel
amin.benmansour.10
<@U05GQD16J1L> has joined the channel