how I built a data collection and uploading pipeline in 2 days with Aureus.

An aside from my normal insane rambling about the human condition, I wanted to write a little technical piece on something cool that I’ve been focusing on.

Astra has always had lots of requests for help from people, and for reasons I’ll outline in a future essay, it became increasingly clear to me that it’s not easy to look for help when you first start doing it, for any reason.

So, as a mini experiment for me to get through my burn out, social anxiety, and large amounts of stress about relaunching literally everything related to Astra, I decided to finish up a half baked codebase idea from forever ago.

Originally Zenith (you might see both names), Everine is designed to be a digital resource database that categories all different charity or government programs someone can access / apply for through the internet.

In 2 days (and probably 7-ish dedicated hours), I finished building Everine’s DB and front-end UI interface for it within Firebase and Aureus, Astra’s open source design system.

Sample image of what a resource looks like in Everine

Defining the resources we would collect

I would say honestly this was the most time consuming part of the whole process, but I found it really therapeutic. I used to work for Multnomah County (aka Portland, Oregon) for one of the data entry teams, and so I was pretty used to the detail and consistently involved.

I had a general idea that I wanted online accessible resources at a very high level, and I wanted it to be a wide variety of things. E.G: We would try to list high level directories instead of listing all of their sub-items. Sometimes this didn’t work, but I felt it would be better to list a directory of all public housing programs in the US as opposed to creating our own.

We had a massive google spreadsheet that matched exactly to the object schema for a resource, as well as using the drop-down to mark which item in an enum was selected. Once the data was filled out, we marked it as green. We only ended up with about 75 resources, but I felt like quality was better than quantity.

The database spreedsheet with all 75 resources on it.

Building the data upload page

Everine’s internal uploading admin screen.

Although it’s not my long term preference due to pricing at scale, Everine uses Firebase Cloudstore. While I was building this project, I finished building a pretty large back end for Faura, where I had to create my first cloud architecture through AWS from scratch. After dealing with VPNs and losing my marbles looking at config pages, I decided Firebase would be an easy start point and then we can see how the traffic does after a while.

The next task was building a UI page for me to get into when I’m running it locally that uploads to the database, making it easier down the line if other Astra volunteers need to access it or if we expand this into an admin app.

The networking code is abstracted, and I tried to follow clean code principals, but I also didn’t put too much thought into it. It took me about 2 days-ish of part time work overall to build, and it plops all of the data nicely into the database Everine pulls from when it goes live in a couple of days.

You can find the code in Github here: Everine.

part of zenith;

//A view that contains the details of a chosen resource.

class ResourceAddView extends StatefulWidget {
  const ResourceAddView();

  @override
  _ResourceAddViewViewState createState() => _ResourceAddViewViewState();
}

class _ResourceAddViewViewState extends State<ResourceAddView> {
  // Takes all of the encoding / decoding keys we use in the JSON conversion process
  // and turns it into a list to be used in our pickers
  var accessTypeData = resourceAccessTypeKeys.keys.toList();
  var detailTypeData = resourceDetailsTypeKeys.keys.toList();
  var categoryTypeData = primaryCategoryTypeKeys.keys.toList();
  var communityTypeData = communityTypeKeys.keys.toList();

  // Text editing controllers to own each text field to access the data later
  var resourceNameController = TextEditingController();
  var resourceDescriptionController = TextEditingController();
  var resourceAccessLinkController = TextEditingController();
  var organizationNameController = TextEditingController();
  var organizationLinkController = TextEditingController();
  var organizationContactController = TextEditingController();

  // Default values for all of the enum types
  resourceAccessType accessTypeSelectedItem = resourceAccessType.browserWebsite;
  resourceDetailsType detailTypeSelectedItem =
      resourceDetailsType.databaseListing;
  primaryCategoryType categoryTypeSelectedItem = primaryCategoryType.crisis;
  communityType communityTypeSelectedItem = communityType.everyone;

  // Creates a resource from the text editing controllers and enum types to deliver
  // to our networking module
  void createResource() {
    var organizationObject = OrganizationObject(
        organizationName: organizationNameController.text,
        organizationLink: organizationLinkController.text,
        organizationContactLink: organizationContactController.text);

    var createdResource = ResourceObject(
        resourceName: resourceNameController.text,
        resourceDescription: resourceDescriptionController.text,
        accessType: accessTypeSelectedItem,
        detailType: detailTypeSelectedItem,
        categoryType: categoryTypeSelectedItem,
        communitiesType: communityTypeSelectedItem,
        resourceAccessLink: resourceAccessLinkController.text,
        organization: organizationObject);

    addResource(createdResource);
  }

  // A layer of separation, sends the resource to our networking module and then
  // resets the page for better UX for me :-)
  void addResource(ResourceObject resource) {
    Networking().addResource(resource, () {
      resetPage();
    });
  }

  // Resets the text editing controllers so all of the fields are blank
  void resetPage() {
    setState(() {
      resourceNameController.text = '';
      resourceDescriptionController.text = '';
      resourceAccessLinkController.text = '';
      organizationNameController.text = '';
      organizationLinkController.text = '';
      organizationContactController.text = '';
    });
  }

  // FACTORY UI ELEMENT METHODS

  Widget createPicker(List<Widget> children, Function(int) onItemChanged) {
    return Container(
      height: size.responsiveSize(100),
      decoration: InputBackingDecoration().buildBacking(),
      padding: const EdgeInsets.all(12.0),
      child: CupertinoTheme(
        data: CupertinoThemeData(
          brightness: palette.brightness(),
        ),
        child: CupertinoPicker(
            backgroundColor: Colors.transparent,
            itemExtent: 40,
            magnification: 1.2,
            diameterRatio: 1.9,
            onSelectedItemChanged: (int index) {
              onItemChanged(index);
            },
            children: children),
      ),
    );
  }

  StandardTextFieldComponent createTextField(
      String hint, TextEditingController controller) {
    return StandardTextFieldComponent(
        hintText: hint,
        isEnabled: true,
        decorationVariant: decorationPriority.standard,
        textFieldController: controller);
  }

  Widget createSpacer() {
    return SizedBox(height: size.responsiveSize(10));
  }

  @override
  Widget build(BuildContext context) {
    // A list of the widgets that hold all of the enum options for our metadata
    List<Widget> accessTypeList = [];
    List<Widget> detailTypeList = [];
    List<Widget> categoryTypeList = [];
    List<Widget> communityTypeList = [];

    // Goes through all of the data from enums and creates widgets for the Cupertino Picker
    for (var element in accessTypeData) {
      accessTypeList.add(Center(
          child: BodyOneText(element.toString(), decorationPriority.standard)));
    }

    for (var element in detailTypeData) {
      detailTypeList.add(Center(
          child: BodyOneText(element.toString(), decorationPriority.standard)));
    }

    for (var element in categoryTypeData) {
      categoryTypeList.add(Center(
          child: BodyOneText(element.toString(), decorationPriority.standard)));
    }

    for (var element in communityTypeData) {
      communityTypeList.add(Center(
          child: BodyOneText(element.toString(), decorationPriority.standard)));
    }

    // TEXT FIELDS --------------------------------------
    var resourceNameTextField =
        createTextField("Resource Name", resourceNameController);

    var resourceDescriptionTextField =
        createTextField("Resource Description", resourceDescriptionController);

    var resourceAccessLinkTextField =
        createTextField("Resource Access Link", resourceAccessLinkController);

    var organizationNameTextField =
        createTextField("Organization Name", organizationNameController);

    var organizationURLTextField =
        createTextField("Organization URL", organizationLinkController);

    var organizationContactTextField =
        createTextField("Organization Contact", organizationContactController);

    // METADATA PICKERS --------------------------------------
    var resourceAccessTypePicker = createPicker(
      accessTypeList,
      (index) => {
        accessTypeSelectedItem = resourceAccessTypeKeys.keys.toList()[index],
      },
    );

    var resourceDetailTypePicker = createPicker(
      detailTypeList,
      (index) => {
        detailTypeSelectedItem = resourceDetailsTypeKeys.keys.toList()[index],
      },
    );

    var resourceCategoryTypePicker = createPicker(
      categoryTypeList,
      (index) => {
        categoryTypeSelectedItem = primaryCategoryTypeKeys.keys.toList()[index],
      },
    );

    var communityTypePicker = createPicker(
      communityTypeList,
      (index) => {
        communityTypeSelectedItem = communityTypeKeys.keys.toList()[index],
      },
    );

    // BUTTONS --------------------------------------

    var addButton = StandardButtonElement(
        decorationVariant: decorationPriority.important,
        buttonTitle: "Add resource to DB",
        buttonHint: "Adds resource to the database",
        buttonAction: () {
          notificationMaster.sendAlertNotificationRequest(
              'Resource created!', Assets.add);
          createResource();
        });

    // VIEW STRUCTURE  --------------------------------------

    var viewWrapper = ContainerWrapperElement(
      children: [
        HeadingOneText("Add a resource.", decorationPriority.standard),
        resourceNameTextField,
        createSpacer(),
        resourceDescriptionTextField,
        createSpacer(),
        resourceAccessTypePicker,
        createSpacer(),
        resourceDetailTypePicker,
        createSpacer(),
        resourceCategoryTypePicker,
        createSpacer(),
        communityTypePicker,
        createSpacer(),
        resourceAccessLinkTextField,
        createSpacer(),
        organizationNameTextField,
        createSpacer(),
        organizationURLTextField,
        createSpacer(),
        organizationContactTextField,
        createSpacer(),
        addButton,
      ],
      takesFullWidth: false,
      containerVariant: wrapperVariants.stackScroll,
    );

    return ContainerView(
        decorationVariant: decorationPriority.standard,
        builder: viewWrapper,
        hasBackgroundImage: true,
        takesFullWidth: false);
  }
}

If I was being particularly picky, I would abstract all of the TextEditingControllers / variables / UI factory patterns and move it into a separate ‘Logic Controller’ file responsible for holding everything outside of the build(context) method. It’s small and easy enough to modify and understand that it’s not a hill I’m willing to die on currently.

The amount of upfront investment moving to Flutter and adopting an org-wide design system has made, really makes things like this possible. Building a to-accepted-standards resource database in less than a week a year or so ago would have been NOT a thing at all, so I’m looking forward to seeing what else it makes for else.

InfoSec for the win, hopefully.

I also wanted to show some more ways that we use Aureus under the surface for internal tooling (like our analytics dashboard that we’re hoping to integrate after all the software relaunches and we have our custom impact tracking framework Senre integreated).

I talk a lot about being a nerd and building software that helps people, and I hope today you got a good idea of what it actually means.