We hosted our second community call on May 31st with a whopping 7 speakers from our sponsored projects Dat, Stencila, and ScienceFair - as well as speakers from the broader community.

It's archived on AirMozilla (thanks AirMoz!) so you want watch it here and follow along with the notes on our Etherpad.

There were over 20 lines of community announcements! Including:

• Decentralized Web Summit 2018 Global Visions / Working Code, July 31 – August 2, 2018 at the Internet Archive, San Francisco, CA https://decentralizedweb.net/
• OpenCon 2018 is coming to Toronto
• Fritter accessible on regular browsers via dat-polyfill and dat-gateway, for more
• Stencila in Binder! (From #eLifeSprint)
• Check out Cabal - decentralized private chat
• PREreview and Dat will be at FORCE2018 in Montreal

We had an incredible line up of speakers:

• Mathias Buus Madsen / Dat & Beaker joined us from Berlin and spoke about Dat and hyperdb, a distributed, scalable database. This is the next big step for the Dat protocol. Hyperdb sets the stage for enabling multiwriter databases. In other words, allowing multiple people to write to a Dat. This is the culmination of many years of work and will enable all kinds of new use cases, including Jim Pick's collaborative shopping list, art, chat, and more!

• Yoshua Wuyts talked about working on a Rust implementation of Dat, funded by the German Prototype Fund. The issue he is addressing is that Dat is written in javascript - so it doesn't run everywhere. Yosh's approach is to port bits and pieces to low-level language. He's 3/4 done implementing hypercore and hasn't hit roadblock - so the experiment is going quite well! The goal is to build a mobile-friendly Dat.

• Hugh Isaacs II joined us from New York. He's made a Dat Chrome extension and he spoke at the last p2p web event in NYC (we'll post the link to his talk when it's available). Hugh is interested in offline friendly tech. Zombie apocalypse example - limited power, limited networking - how do we connect in those types of situations? Even in NYC, disasters like Hurricane Sandy show the importance of being ready with offline friendly tools. Hugh is focused on trying to get the word out there and highlight the ability for protocols like Dat to work offline.

• Naomi Penfold, PhD just ran a productive and inclusive hackathon - The eLife Innovation Sprint - in Cambridge, UK. Joe and I both went, and we invited Naomi to the call to talk about her experience planning and running the event. She flipped the Q and A by asking our audience questions in the Etherpad (line 243), including "How can we reach more under-represented people in the tech space?" Review her extensive notes on planning and executing the event, and reach out to her with your comments!

• Georgia Bullen and Chris Ritzo joined us from Measurement Lab to tell us about the Open Internet Measurement Platform -- and yes, folks, it's all open source, open data. Measurement Lab's mission is to measure the internet, save data, make it easy for people understand. To do this, they partner with Open Tech Institute, Planet Lab (Princeton), and Google. Since 2009, they've been working to provide OS platform to host measurement tests. This means you can ask "how fast is my connection?" and get a real answer for your actual house. They collect data from millions and millions of measurements per day. Right now they're working on a project to with colleagues at Simmons College & Internet2 to make better system for libraries - learn more at Chris' slide deck. Reach out to Measurement Lab and join their mailing list!

• Nokome Bentley / Stencila joined us from New Zealand to talk about what's new with Stencila. The team has released Stencila Desktop 0.28.0 a few weeks ago for beta testing - check it out! They've also started a beta testing program, learn more here and reach out to get involved. Daniel Nüst and Min Ragan-Kelley created nbstencilaproxy for running Stencila inside Binder during the eLife sprint, which was a lot of fun for everyone. Nokome has been simplifying the approach to writing and registering custom functions in Stencila. He hopes to make this part of the next release.

• Rik Smith-Unna / ScienceFair joined us from the UK to tell us about "The futuristic, fabulous and free desktop app for working with scientific literature 🦄". Right now ScienceFair is a desktop app for discovering, collecting, organizing, reading and analyzing scientific papers. Long-term vision for it is a complete rethink of how scientific literature is produced, distributed, discovered and used. Works p2p, built on dat (migrating to hyperdb). Stay tuned for bug-fix releases and more in the coming months! Join the ScienceFair community on Gitter.

Thank you to all our speakers, AirMoz, Aurelia Moser, and to everyone who tuned in and asked questions. See you again in August. In the meantime, please reach out if you'd like to speak on a future call @codeforsociety on Twitter or by email at hi@codeforscience.

The last call was super fun too, check it out!

Research and cultural heritage institutions are facing increasing costs to preserve digital objects like scientific data, digital art, and other artifacts. As many institutions move data to cloud services, preservation costs and complexity are quickly becoming concerns. We are announcing a project to prototype shared infrastructure for digital preservation: Cooperative Preservation Network.

Together with other leading institutions in decentralization and digital preservation, Code for Science & Society, the Internet Archive, and California Digital Library, we aim to demonstrate how decentralized technology can bolster existing institutional infrastructure. Built on a decentralized network using the Dat Protocol, this project aims to enable public organizations who preserve digital cultural heritage to backup and monitor digital assets.

Dat is already being used by researchers, developers, and artists. This project aims to identify institutional limitations to using decentralized technology, whether technical and social. To test our assumptions, we will prototype a network allowing each participating entity to view and download the collections of other participants. If successful, this project will demonstrate how to reduce preservation costs while increasing preservation assurance, as members of a cooperative, decentralized network mutually support each other to ensure adequate copies of data are maintained.

Open infrastructure allows institutions to use digital preservation tooling without locking them into specific paid services or locking data into a patchwork of data silos. By working with the Dat Protocol, we will build this project to maximize flexibility and interoperability. Our goal is not to replace existing institutional infrastructure but to make it more capable by linking institutions at a foundational level. Building on value-driven open infrastructure, this project aims to identify new opportunities for collaboration between institutions and community engagement in data preservation.

Moving Forward

Despite improvements in data preservation and access, today’s digital preservation solutions rely on storage of objects in centralized servers. This model is built on traditional web infrastructure, which was designed with the values of commercial organizations. It’s time for scholars to ask whether today’s data preservation technologies align with open scholarship’s values of access, preservation, privacy, and transparency.

This project will be a community-driven infrastructure that values openness and bakes access into the code. Want to learn more? Representatives of this project will be at FORCE 2018, Joint Conference on Digital Libraries, Open Repositories, DLF Forum, and the Decentralized Web Summit.

More about CSS: Code for Science & Society is a nonprofit organization committed to building public interest technology and low-cost decentralized tools with the Dat Project to help people share and preserve versioned digital information. Read more about CSS’ Dat in the Lab project, our recent Community Call), and other activities. codeforscience.org

More about IA: The Internet Archive is a non-profit digital library with the mission to provide “universal access to all knowledge.” It works with hundreds of national and international partners providing web, data, and preservation services and maintains an online library comprising millions of freely-accessible books, films, audio, television broadcasts, software, and hundreds of billions of archived websites. archive.org

More about CDL and UC3: University of California Curation Center (UC3) at California Digital Library (CDL) provides innovative data curation and digital preservation services to the 10-campus University of California system and the wider scholarly and cultural heritage communities. Learn more about UC3’s collaboration with CSS in our previous Dat in the Lab project. https://www.cdlib.org/

We've been traveling so much that we are a little late on the announcement (eek it's next Thursday!) but please mark your calendars for the next Code for Science & Society Quarterly Community Call.

Thursday, May 31, 2018: 11am Pacific / 2pm Eastern / 7pm UK / 8pm Berlin / June 1, 6am NZ

We have an incredible line up of speakers:

• Yoshua Wuyts
• Naomi Penfold, PhD
• Hugh Isaacs II
• Georgia Bullen / Measurement Lab
• Mathias Buus Madsen / Dat & Beaker
• Nokome Bentley / Stencila
• Rik Smith-Una / ScienceFair

The last one was super fun, so you won't want to miss this.

How do you watch? Thanks to Aurelia Moser at Mozilla, you can tune in on Air Mozilla and ask live questions on our Etherpad.

To recap:

• May 31, 2018: 11am Pacific / 2pm Eastern / 7pm UK / 8pm Berlin / June 1, 6am NZ
• Read along, add your updates, and ask questions here: https://public.etherpad-mozilla.org/p/CSS-community-call-May-31-2018
• Watch the call here: https://air.mozilla.org/mozilla-science-lab-may-2018-bi-monthly-community-call/
• Can we feature your work on an upcoming call? You better believe we have a spreadsheet going, so email us hi@codeforscience.org to get on a future call.

This photo of Joe explaining deeply exciting computer stuff is from the eLife Innovation sprint, taken by Julieta Sarmiento

At the end of April we joined scientists working on Chan Zuckerberg Science's (CZI) Human Cell Atlas (HCA) project to facilitate a session on collaboration for scientists. Improving communities ability to collaborate is a core focus of the Code for Science & Society mission. We facilitate in-person events, share our processes openly, and join community groups like the Open Source Alliance for Open Scholarship and Joint Roadmap for Open Source Tools to advance this mission. Our unique perspective on collaboration and open source comes from work with projects and people across domains from science to the arts. What does this facilitation process look like? Read on to find out!

A successful collaboration depends on people and communication. Learning how to leverage tools like Git is part of the process. To develop a sustainable collaborative process, we work through a blockers and concerns to create a project-specific method for collaboration. Our process looks a little different each time and is shaped by the needs of the group.

Facilitating Collaboration with Git for Computational Biology

At the CZI HCA event, we worked with research scientists for two hours with the goal of improving their collaborative processes. The participants represented a mix of computational biologists, bench biologists, students, postdocs, and faculty. Most of the people at the session were familiar with Git but not regularly using it for collaboration. It's often easier to learn Git than it is to develop a Git-based workflow for a group. The basics of Git can be covered in a day. Developing a lab workflow that is sustainable takes communication, iteration, and commitment.

Starting with terms and concepts

To ensure a successful session, we make sure everyone is comfortable with the terms and on the same page. We began with a terminology review of Git. This helps to raise a questions (e.g. when to make a branch vs when to fork) that we return to later. We also reviewed the role of common files, such as a README, that help set a project's norms and expectations.

Motivations and blockers

After discussing terminology, we to motivations and blockers. We wanted to surface attendee motivations and blockers in team collaboration. To get at this information, we asked attendees to answer the following questions:

• What's the easiest part of your daily collaborations?
• What do you want to improve?
• What is keeping your team from using a Git-based workflow?

What's easy?

To identify and create standardized communication and software development practices for their labs, we started with what's working. We ask the question "What's the easiest part of your daily collaborations?" to start participants off thinking about what works for them now. Later, when we discuss blockers, we can return to "what's easy" and re-frame the blocker in terms of what the participants have identified as easy for them. We saw answers such as:

• in-person conversations
• regular communications they have with their lab
• when I know who is responsible for what
• we know who needs to make decisions
• communication is clear

Two categories of answers came up. Conversations with people and situations where expectations are clear are the core of well-done group collaboration.
(One participant who runs a research group brought up SCRUM, see Adapting Scrum to Managing a Research Group for more.)

What needs work?

By asking "What do you want to improve?" we looked for common themes motivating participants to attend the session. Most of the participants already knew how to use Git. However, a knowledge of Git does not guarantee a smooth collaboration and a successful project.

There were two categories of responses: technical skills and project management. Some participants wanted to improve their working knowledge of Git, for example a person who understands branches but is not sure that they're doing right and is looking for best practices. Other participants wanted to improve their team's approach to project management and were interested in implementing code review, standups, and other processes to keep the team working together efficiently. Some of the answers were:

• get a handle on branches, forks, merges
• get everyone on team aligned doing things the same way
• getting feedback regularly
• code review

Code for Science and Society has put a lot of effort into understanding and developing systems for diverse teams. We've seen that the best process tends to be the one that can be adopted and sustained. In our discussions of best practices, we surfaced some key points, reviewed open source best practices, shared community resources like openopensource.org, and discussed how to use issue templates.

Blockers

With a view of what was easy for participants and what they wanted to improve, we then moved on to discuss the specific blockers. We did this by asking, "What is keeping your team from a Git-based workflow?" Interestingly, the main blockers were things we classify as people problems rather than technical problems. Examples included:

• people in my lab have different levels of understanding of Git
• communication — lack of common language, lack of Git familiarity
• some people don't like change
• students have projects of their own and don't have collaboration needs

While most of our participants had a good understanding of Git, not everyone in their work groups are at the same level. This creates situations where there is no common process or understanding. Fortunately, team members don't need a deep understanding of Git to collaborate. People who are not familiar with Git can use the GitHub web interface to submit issues and comments, contribute to project management with GitHub Projects, and write documentation and user guides. This is critical to helping those people develop comfort with GitHub and incorporating them into a Git-based workflow, even if they won't be making commits for a while.

The last two points raised by participants fall into the "we don't need to bother with this" category. Some people don't like change. However, we have seen people who are resistant to change get on board quickly once they see a project progressing. An open source style workflow allows anyone in the group to see project progress, and this tends to have a motivating effect on supervisors and others.

The flip side of "we don't need to bother with this" is a scientist who is used to working independently. In most groups there is an expectation that their code and methods will be reused by future lab members. Even an independent project can benefit from code review. We encourage people to onboard independent workers with a code review process that formalizes and documents the comments that are already happening in the lab. (See this Gist on code review and scientific review in labs. See also Mozilla Science's Code Review in Labs)

Initiating a standard collaborative process

The majority people who attended this session were looking for advice on best practices that were specific and relevant to their work and processes. There's no magic recipe for a good collaboration. Attendees were able to connect with other groups who collaborate with Git. At an event like the CZI HCA meeting, the opportunity to make connections with other groups working towards better project management on similar projects is valuable. Finally, by creating a space for honest discussion, we set up groups to evaluate their existing project management workflows, surface and discuss blockers to best practices with their labs, and push for improvements.

We're always interested in working with groups to bring the best of open source workflows to projects in new domains. If you're interested in working with us — reach out! hi@codeforscience.org

Over the last year, we've been developing HyperDB, a distributed scalable database for peer-to-peer collaboration. To demonstrate HyperDB, I built Dat Shopping List — an application that makes grocery shopping fun again!* Along with being a powerful decentralized key-value database, HyperDB will be integrated in Hyperdrive to allow for multi-user collaboration with Dat archives. We are really excited about the potential of this and working on integration over the coming months. Read more to learn about the most decentralized shopping list ever built!*

*wild claims may not be accurate

Dat works without centralized servers. (Credit: @mafintosh)

Dat is a purpose-built tool to allow sharing of research data in a way that enables effective collaboration and reproducibility. Out-of-the-box, it gives you a distributed network, data history, and security. With the development of HyperDB, we are excited to see Dat growing to the foundation for other community collaboration tools.

Dat is growing and evolving

Since its initial release, Dat has attracted an enthusiastic developer community with ideas and use cases that extend beyond its original mission. Dat has evolved into a whole ecosystem of related open source tools, libraries, and services. Some examples:

• the dat command line tool for sharing data (online tutorial)
• the Dat Desktop app for sharing files over the internet
• a full ecosystem of open source libraries in JavaScript (plus a new Rust project is growing fast)
• commercial services such as Hashbase are providing hosted solutions
• Beaker Browser is a powerful vision of a web browser that gives all people the power to publish their work to the web without servers
• the community has even organized grassroots Peer-to-Peer Web conferences in Los Angeles and Berlin, and soon New York

Tara Vancil giving a talk at Peer-to-Peer Web L.A. (Credit: @jimpick)

Dat is designed to evolve. The initial version of Dat was designed to allow a single researcher to publish a set of files to the world, and to be able to update the files over time ... all while preserving history for reproducibility. But quickly, Dat caught on with other use in a variety of communities.

HyperDB: Dat Multiwriter

One of the most desired features for Dat is collaboration between multiple users, known as multiwriter (as opposed to singlewriter, which Dat is currently). We especially heard this need in the feedback from the research community. Led by Mathias Buus (@mafintosh), we have spent more than a year developing HyperDB, a distributed key-value database. Alongside this, we are upgrading the core internal libraries of Dat to use HyperDB and support improved collaboration. HyperDB is working today, but we are continuing to test and integrate into hyperdrive and other Dat tools.

Multiwriter support is a major change. To ensure a smooth roll out, we are working with the Dat Protocol Working Group to define backwards compatibility requirements. This process, and work on integration will be a priority in the coming months.

Dat Shopping List Demo

The Dat Shopping List application demonstrates how to use HyperDB to collaborate with other users on a single database. We built this to showcase multiwriter support and guide others on its use. Code for Science & Society, the non-profit organization sponsoring Dat, contracted me (Jim Pick) to demonstrate how HyperDB works, improve any API peculiarities, and surface any remaining bugs.

You can launch the demo in any web browser:

• https://dat-shopping-list.glitch.me/

A quick preview of what you can do with the app

The demo is a simple to do list app, in the spirit of the TodoMVC project.

Video Walkthrough

Here is a short (2.5 minute) walkthrough of the demo.

Mobile phone support

Where the demo really shines is on your mobile phone. The demo is built using some cutting-edge web browser features, so you can use Add to Home Screen on your phone, and you can use it just like you would use any other app.

Offline support

When you create a shopping list, under the covers, you are actually creating a multiwriter Dat archive. Each shopping list is a separate Dat archive, and each shopping list item is just a tiny little file inside a directory.

The master copy is stored in the storage of your web browser, along with the secret key which ensures that you are the only one that can modify it.

Having the master copy inside your web browser is great, because that lets you make changes to the document, even when you aren't connected to the internet. So you can put your phone in airplane mode, and still use the app.

Making changes anytime is okay, even when offline. Changes sync when going back online.

Synchronizing and Keys

Of course, a multiwriter demo would be boring if it only ran on one device. Dat has always had great support for replicating data across the internet.

Dat has been single writer up until now, and it's identity model was essentially a single key pair, a set of keys — one public key and one private key. With the new support for multiple writers, each writer will have its own key pair, and users exchange keys to grant access to new writers.

In this demo, the user shares their keys to the shopping list owners to get authorized.

We authorize new writers by copying and pasting their keys into the owners shopping list.

Once authorized, other users can update the shopping list.

A truly user-friendly application might hide the keys and key exchange details from the end users by incorporating some sort of identity-based account and login system. But, for this demo, we wanted to illustrate the core authorization mechanism that multiwriter Dat uses. Asking people to understand and do the key exchange manually is not very friendly, but it is flexible and powerful.

Node.js Gateway

This demo works in any modern browser. It has been tested in Chrome, Firefox, Safari and Microsoft Edge, as well as Mobile Safari and Chrome on Android.

If the demo had been written for Beaker Browser, which speaks the Dat protocol natively to the internet, there would be no need for a gateway. But in order to allow the demo to work across many browsers, the app replicates the shopping list data across a websocket connection to the server from which it was loaded. The Node.js server will then talk to the rest of the peer-to-peer swarm on behalf of the client.

By necessity, the data is shared with the gateway, so if you want to share data privately, you will need to run your own private gateway. It is not as scary as it sounds though... all the code is on GitHub, and you can remix the demo on Glitch with a single click.

Podcast Interview with Mathias Buus

If you would like to learn a bit more, Mathias was a recent guest on the DatCast podcast and talked a bit about the origins of the Dat project and how multiwriter and hyperdb came to be.

Support the Dat Project!!!

Thank you to Code for Science & Society for hiring me to build the demo. They are a non-profit and depend on contributions to continue running. Donations are greatly appreciated.

Data intensive research depends increasingly on the sharing and reuse of datasets. Unfortunately, collaboration can be stalled by technical roadblocks around sharing large datasets between institutions. Today, sharing files between institutions remains challenging without extensive technical infrastructure or external services.

As a part of the Dat in the Lab project — a project done in partnership with California Digital Library and funded by the Gordon and Betty Moore Foundation — we have worked with University of California researchers to better understand barriers in their work. This is part of our participant-driven design process and our goal is to identify where we can remove blockers for people in data intensive fields with Dat-based workflows. What have we learned? Read on!

At the start of the project, one issue came up immediately — researchers still struggle to share large datasets between institutions. Why is this? Many commercial or open platforms exist for transferring files. But many of these tools require file transfer through a centralized server. By adding a third party, the act of transferring a file becomes more complex, more expensive, and reduces security. Dat works by linking the two people directly, without needing a third-party server. However, as anyone who's worked in institutional computing knows, connecting two institutional networks is not always straightforward. As we learned, institutional network security does not always mix well with peer-to-peer protocols.

We want make Dat great for researchers, but many researchers lack access to install software globally on their systems. With that in mind, we also created a new simplified install process and improved the feedback in the Dat command line tool to make it more intuitive. Our new installation method makes it easy to install Dat without sudo or npm, in just one step!

We've always aimed to make Dat great at data transfer. But sitting down with researchers and walking through each step showed us where we needed to improve. Dat in the Lab has provided new chances to try Dat out in new contexts, with folks at various levels of experience. From these experiences, we have made improvement that will make Dat easier for everyone to use.

Case Study: UC Davis to Washington State University Data Transfer

Here’s a short summary of a recent data sharing use case we facilitated. We've designed a workflow to enable researchers to share data across institutions, from easy installation to direct data transfer.

For Dat in the Lab, we helped a researcher from UC Davis securely share pre-publication genomic data with an external collaborator based in Washington State. At UC Davis, PhD student Ryan Peek is sequencing Sierra yellow-legged frogs (endangered species, Rana Sierrae) and Meghan Parsley, a PhD student at Washington State University (WSU) wants to collaborate on Ryan’s data.

In the end, we successfully helped Ryan and Meghan to send the data from Ryan’s server at Davis to Meghan's laptop at WSU. Ryan’s genomics group at Davis had a server that we installed Dat onto using the new standalone installer method. Ryan used the dat share command (in the folder where his data was located) to start sharing his research data and sent the Dat link to Meghan.

Initially, we tried to transfer the data onto a server on the internal network at WSU. We immediately ran into connection problems, where Dat was unable to connect to external networks. To troubleshoot this, we ran dat doctor on both ends. Dat doctor tests general connectivity and provides a command to test peer-to-peer connectivity. The Dat doctor determined that the server at WSU was not able to connect directly to Davis, most likely due to institutional firewall settings. We are always try to make the doctor better at spotting networking issues that may prevent Dat from working!

To get away from the institutional firewall, we instead transfered the data to a researcher's laptop. It worked! Ryan and Meghan were able to exchange data directly between their computers. This transfer did not require any external infrastructure or costly platforms. Additionally, Meghan was able to close her laptop and have the data transfer automatically resume when she was back online. Dat provides researchers with flexibility for quick file transfers with powerful features underneath for users that need more advanced features.

Work Flow — Dat For Data Transfer

Interested in beta testing Dat for secure data sharing? Try out the data sharing workflow with a friend and let us know how it went!
Follow this workflow and share your experience at hi@datproject.org, especially if you work as a researcher. We want to know if you succeed, or get stuck, or have an thoughts or feedback. We're interested in hearing from you, so don't hesitate!

1. Install the Dat command line tool on two separate computers that you want to share data between (can be both yours, or it's more fun with a friend).
2. With Dat, one person shares a folder and the other downloads. On the sharing side, cd into the folder of data you'd like to share and run the command dat share. It will print out a long link that starts with dat://.... If you want to cancel the share at any time, type CTRL+C. Your data is also private, so nobody can discover or access your data without giving the Dat link to that person.
3. On the downloading side, run the command dat clone dat://… with the link you received. This will create a new folder where the data will be downloaded.
4. During the transfer, you should see connection statistics and progress bars on both computers.
5. If the transfer fails or never starts, send a screenshot of the output of the command line to hi@datproject.org so we can understand where users are getting stuck.
6. If the transfer succeeds, you successfully distributed your data over a secure peer-to-peer network!

Ok — how’d it go? We'd love to hear what you think, email and tell us about it. We're also interested to chat about potential use cases for Dat in your research workflow!

Want to play around some more? Try some more advanced topics over at try-dat.com. (Note this tutorial starts with the npm installation, you can npm install or one line install.)

The following post is based on my talk at the library and technology conference Online Northwest in March 2018, titled “A vision for decentralized data preservation across a network of libraries and trusted institutions." Photo credit to Sarah Seymore. I'll update with the link to the recording when it's available.

Technology is imbued with the values and biases of its creators (see the work of Safiya Noble for more). Today’s online data storage and preservation systems are no exception. They are built on traditional web infrastructure, which was designed with the values of hierarchical and commercial organizations. It’s time for scholars to ask whether today’s data preservation technologies align with open scholarship’s values of access, preservation, privacy, and transparency. Alternative communication tools, such as email and RSS, were built for interoperability and portability but are now largely relegated to (small) uses. Similar approaches built with decentralized technology have recently reached a new level of maturity and publicity.

It’s time for scholars to ask whether today’s data preservation technologies align with open scholarship’s values of access, preservation, privacy, and transparency.

Decentralized tools offer a more robust, open alternative for data management. At a fundamental level, decentralized systems distribute data across a network of linked participants. Beyond scholarly use cases, decentralized models are also changing the way the web is built as artists, activists, and technologists use them to rethink the web. This technology offers new models for community managed information sharing on the web. As decentralization remakes the legacy web, it presents scholarship with an opportunity to rethink who owns scholarly data and the pathways to access that information.

Today, decentralized approaches like the Dat Project offer foundational tools for integrating isolated data silos, operating at a lower level of web infrastructure to link information in existing systems. (At the end of this post is a plain language Dat Project explainer.) These modern tools present opportunities for scholars, librarians, and technologists to redesign long-term data preservation in a way that formalizes the shared values of the community within the technology itself.

The Internet is broken and we are using it to access and distribute all of human knowledge.

Today’s web is dominated by centralized, non-interoperable entities that sustain their businesses by enclosing and selling data we provide to them. The model of controlling content via an online platform has been implemented in every industry, from social media to scholarly publishing. We live in an age where data are increasingly collected, analyzed, and repackaged for sale. Across domains, data live online (I mean data in the most inclusive terms): the work of a writer, government data, newspaper archives, your family photos, scientific data, film archives, artist’s creations. These data live on the web with varying degrees of strategy, management, and plans for long-term upkeep.

The lines drawn around which data have political value, research value, or business value constantly shift. Scholarly data - the majority of which are presented, accessed, and stored online - are not an exception, and can move quickly from niche to politically charged (see DataRescue). Although data are valuable, data management and stewardship practices are extremely inconsistent and financially burdensome. Link rot (when links break) and content drift (when the information at a link changes) plague fields from legal judgements and scholarship to biomedical research. Together link rot and content drift create “reference rot” — you can read more about the impact of reference rot on scholarship across scholarly disciplines here, here, and here.

In this landscape, librarians, technologists, and scholars are trying to manage systems that will sustain the preservation of humanity’s growing knowledge base forever. The web is not designed for long term preservation of information. As Laurie Allen said in 2017: “the internet is a terribly unstable way to keep information available.” As decentralized models to connect people to information are developed, these models present a unique opportunity to rethink how scholarly data are stored and accessed online.

A network approach

Centralized data storage systems can only preserve what they hold in their servers. These models require custody to provide access. Data custody becomes increasingly expensive and difficult to manage as data volumes increase. Stephen Abrams asks the question, “can we replace custody with easy access?” In other words, is knowing where data are, and trusting the preservation standards of that location, equivalent to (or better than) custody? Can we reduce the burden on institutions to own everything with a mandate to know where data are and how many verified copies exist?

The idea of “preservation in place” where libraries bring “preservation services to the content” is not new. In a decentralized model, custody is not required for access. By bringing preservation services to content, we replace custody with access and preservation. Data then live in a network of linked institutions that model a commons of trust.

To create a functional system wherein information is shared across silos between trusted institutions is a utopian idea. What would such a network require beyond technology? The most critical factor is trust. Trust in each participating entity’s standards and processes. Trust between institutions. Trust from the community that a commons can be sustained without tragedy. Today's decentralized models make preservation in place technically feasible, and interoperable with existing data preservation silos. But perhaps the cultural part of modeling a new system will be more challenging than writing the software? ;)

Join us

Scholarship today is another form of online content creation. I resisted the idea for years. But the parallels are clear. Scholarly work is used by for-profit entities to drive clicks, bring advertisers and subscribers, and sustain their businesses. We give up custody of — and sometimes rights to — our work. We then must pay for access to our own data and publications, or lose access to those resources. I believe this system is unsustainable because it is not based on the community’s values. Reducing reliance on centralized systems will help to return the control of scholarly assets to the creators and trusted institutions that value access and scholarship. Data preservation and access are fundamentally about trust. Who do you trust to steward human knowledge? I trust libraries, scholars, and public interest technologists over entities with clear business interests.

What’s important to the scholarly community? Are those values are reflected in our technology?

Today’s online scholarly infrastructure values systems of centralized control, people with reliable internet connections, people/institutions with money to pay for access (when there’s no money — it values people/institutions with the technical capacity and time to do the work themselves). The web is being reimagined today as a network wherein data are freely shared between linked users. For scholars and librarians, it’s a chance to step back and assess what assumptions and values are baked in to today’s open scholarly infrastructure. What’s important to the scholarly community? Are those values are reflected in our technology? Let’s reexamine how scholarship and data live on the web and move into a future where our values are reflected in our technology choices.

Save the date for our next community call May 31st 11am PST. Join our mailing list for a reminder. Follow me us on the tweets, @daniellecrobins, @dat_project, and @codeforsociety.

Thanks to the 2018 Online Northwest Program Planning Committee for inviting me to speak and organizing such a fantastic conference! And extra thanks to Joe Hand, Karissa McKelvey, John Chodacki, Robin Champieux, and Stephen Abrams for great discussion and comments on drafts of this work.

What’s Dat why are we working with this technology?

Dat is an open source, non-profit backed peer-to-peer file sharing protocol originally developed to distribute large datasets. It’s not blockchain based, and instead uses an append only log to track changes and a private key to allow an author to make changes.

When a folder is tracked with Dat, it creates a unique persistent identifier for that package of data (whatever is in the folder). This unique identifier is not based on location or the folder’s content. In this system, a folder of data can change location or contain dynamic content while keeping the same identifier. Dat then tracks changes to the contents of the folder with a transparent change log. Any reader can view the change log, early versions of the dataset, or keep the folder synced to always have the latest version.

The Dat identifier can be used to track that package of data across a network. Using the identifier, anyone can see how many verified copies exist, download copies, and re-share it from another computer. A Dat package can contain any file type and tracking with Dat does not change the contents of the package. It’s a lightweight and flexible system that prioritizes user control of data sharing across a decentralized network. For more on how Dat works, check out the whitepaper.

We are thrilled for Code for Science & Society to be a host organization for the 2018 Ford-Mozilla Open Web Fellowship. Our Open Web Fellow will work on using Dat and peer-to-peer technologies for cultivating a healthier internet for the public good.

What does this mean? You can apply to work with us, or one of the other great host organizations, for 10 months starting in September 2018! You'll get paid! And health insurance and childcare supplements! You'll go to MozFest! You'll meet so many people that your world will expand.

At Code for Science & Society (CSS) and the Dat Project, we believe that community-driven, decentralized technologies are the future of the web. Today, the rise of centralized services threatens freedom and privacy. Decentralized, open alternatives exist, but are not yet widely adopted. Dat is an open source peer-to-peer protocol. It’s the basis of peer-to-peer browsers (Beaker Browser), social networks, and data sharing tools. Backed by CSS, Dat takes a user-centered approach to developing projects. Hackers, activists, and champions of the open web have been drawn to the Dat community where they are creating a new ecosystem of peer-to-peer applications.

Key details

Application link: https://foundation.mozilla.org/fellowships/apply/

Application dates: The fellowship application opens today(!) and closes on April 20th (Friday at 5pm ET).

What are we looking for in a fellow? We are looking for someone who will make an impact on the peer-to-peer web over 10 months. We are a small, flexible organization and we are looking for a good fit. The fellow has the opportunity to select a focus area of their choice related to our work.

Does the fellow have to work with Dat? The Dat Project is our flagship sponsored project, and much of our community and resources center around it. We are looking for a fellow who will make use of our existing strengths, which will probably involve working with Dat in some way. You do not need to be an experienced Dat user or developer to be a good fit for this fellowship.

Location, citizenship: A fellow can be based anywhere and will work remotely! Danielle and Joe, will mentor the fellow. We are both located in Portland, OR, USA. The Dat team is distributed across the world. If you want to come work with us in Portland for some (or all) of your fellowship, that works too! Just be aware that we don't have an office, we work out of our homes and coffee shops. The coffee is good here!

Application logistics: There are a few different fellowship options, select "Open Web" to apply to work with us.

Questions?: Reach out to us (hi@codeforscience.org, @codeforsociety) or contact Mozilla (fellowships@mozilla.org) with any questions. Also keep an eye out for the Mozilla AMAs on the application.

Last week we brought together people from across industries including academia, publishing, and technology for our first community call. We will be hosting these calls every quarter to spread ideas, make connections, and learn about each others projects. As a nonprofit, we work to improve access to knowledge across several domains and this call is a perfect venue for facilitating that.

Did you miss it? You can watch the the recording and view the notes! Follow Dat Project on Twitter or sign up for our email list so you can catch the next one.

Why host a call? We wanted to add another channel for regular project updates and a chance to make our online connections a little more IRL. We have a unique community! We can pull speakers from friendly and interesting projects across multiple domains who may not cross paths but face similar challenges. It's a chance to bring the global community together to interact between conferences and meetups.

We currently support three projects via fiscal sponsorship and are growing an awesome community of technology for the public good projects that span open data, open research, science, peer-to-peer web, and technology for the public good. Central to our mission to bringing people together from different fields to share ideas.

During the call, we hosted three speakers, updates from our sponsored projects, and non-verbal updates from the community:

• Karissa McKelvey (one of our fantastic new board members!!!) and Stephen Whitmore spoke about Digital Democracy. Digital Democracy’s mission is to empower marginalized communities to use technology to defend their rights. In simple language,they fight for indigenous self-determination through tech & local partnerships. They build tools that work in remote, rural locations that help people defend their land ownership and other rights.

• Peter van Hardenberg did a demo of PixelPusher. An experiment to really understand the limitations and needs for peer-to-peer applications and data sync. You really need to check out his demo! It was pretty cool to see real-time how collaborative conflict resolution can happen on via a peer-to-peer network.

• Tara Vancil covered Beaker Browser, a project near and dear to our hearts. If you are following Dat and haven't seen Beaker, you are in for a treat! With Beaker you can browser websites over Dat and so much more! Tara gave some updates and vision of where they are headed.

• Danielle and Joe did the Dat update, calling out recent work on Dat in the Lab. Stay tuned for an update to the CLI to address data transfer issues for researchers. The new Dat Protocol Working Group is working to formalize existing Dat specification and create processes for updating protocol. This group meets every other Wednesday in #datprotocol IRC - all welcome!

• Nokome Bentley delivered the Stencila update. Stencila aims to make reproducible research more accessible through familiar word processor and spreadsheet interfaces. Check out the (most recent builds)[http://builds.stenci.la/stencila/]. Stencila is also looking to develop function libraries, to make it easier for users to add custom functions and to write domain-specific function libraries that can be used within Stencila. Learn more at the Stencila work in progress libtemplate. And finally, on the interoperability front, chek out Stencila's ongoing work on converters. Initial working version of converters for Jupyter Notebooks, RMarkdown to Stencila Articles (so you can port your markdown notebook and code into a JATS-based article format to write up a publication) are just one exciting thing discussed!

• ScienceFair's Richard Smith delivered an async update. He's partnered with WorldBrain to work on shared distributed knowledge infrastructure issues. In the process, he's developed blazing fast Dat compatible fulltext search index with WorldBrain in pure JS that scales linearly to millions of documents (expect release in the next month - boom! 🔥🔥🔥). Major work on ScienceFair datasources is underway, including all of PMC. This would mean that theree full-text archive of biomedical and life sciences journal literature maintiained by the U.S. National Institutes of Health's National Library of Medicine (NIH/NLM) would be searchable, annotatable, and sharable. Stay tuned!

A few events were announced on the call:

• Danielle Robinson will keynote Online Northwest, a conference on libraries, technology, and culture - March 30th in PDX
• Tara Vancil (Beaker) and Mathias Buus (Dat) will speak at SXSW - Democratizing Data Science with Offline First - Mar 10, 2018
• Paul Frazee (Beaker) will be speaking in London at a Tech We Love event - Mar 21, 2018
• May 10- 11 2018 - eLife Innovation will host a sprint in location in Cambridge to coincide with the Mozilla Global Sprint. If you join the Mozilla sprint to build something with Dat and/or Beaker be sure to tell us about it!

Thanks to all our speakers, attendees, and extra special thanks to Aurelia Moser and Steph Wright (and the Mozilla Science family) who helped us get organized and live stream the call.

• Recording: https://air.mozilla.org/code-for-science-society-community-call/
• Notes: https://public.etherpad-mozilla.org/p/CodeforScienceandSociety-Community-Call-2018-03-01

Follow Dat Project on Twitter or sign up for our email list to get notified for the next community call.

Have a project that you want to share on the next call? Let us know! Email hi@codeforscience.org with information about your project and why our community needs to learn about it.

Save the date for the first ever Code for Science & Society Community call!
Join us March 1, 2018: 11am Pacific / 2pm Eastern / 7pm UK / 8pm Berlin / March 2 8am NZ!

Featuring updates from the Dat project, Stencila, and ScienceFair. As well as speakers from the community. Karissa McKelvey will talk about Digital Democracy's Mapeo is an offline-first peer-to-peer desktop app for Open Street Map. It uses underlying pieces of Dat for it's data and replication models! Peter van Hardenberg will talk about modeling merge conflicts with a pixel art editor. Team Beaker Browser will update us on their work.

Our agenda is evolving here: https://public.etherpad-mozilla.org/p/CodeforScienceandSociety-Community-Call-2018-03-01

Do you want an email reminder?

Our goal with the community call is to add one more place for the people to connect with our community of projects and people. We will feature CSS sponsored projects and speakers from the community talking about their work. It's a chance to bring the global community together to interact between conferences and meetups. It'll be fun.

We are still confirming the logistics, but the call will be streamed! We will update this post with information on how to tune in.

Do you want to speak on a future call? Are you doing something in the public interest technology space? Approaching a technical problem? Or did you make some weird/fun art we should check out? Want to share with our community? Email Danielle and Joe hi@codeforscience.org!

We held our first board meeting in San Francisco last week where we welcomed two new board members and a board advisor. Our board oversees operations and will support us as we develop sustainable strategic vision for Code for Science & Society (CSS) and the Dat Project.

The role of CSS, a nonprofit, has been two-fold, it has housed Dat leadership and acted as a fiscal sponsor for other projects (learn more at codeforscience.org). As we formalize the structure of CSS, we will ensure Dat governance is in line with open source best practices and the community's needs. CSS will continue to employ people involved in Dat Project, but new community-rooted processes will govern Dat and bring more transparency to the project. We will communicate more on the process of establishing a new governance model over the coming months. In the meantime, we’d like to introduce our board:

Kaitlin Thaney is the Endowment Director of the Wikimedia Foundation and is our new 🎉 Board Chair 🎉!!! Her work focuses on long-term sustainability and support for the free and open knowledge movement. Prior to joining Wikimedia, she oversaw the Mozilla Foundation's network of community programs, a $10 million portfolio crossing science, education, Internet of Things, advocacy, and gender-based programming. She also worked to build programs such as the Mozilla Science Lab, Digital Science, early Open Access and Open Data work with Creative Commons, and Datakind UK. Kaitlin was formerly on the Dat project advisory board. Kaitlin was elected to the position of Board Chairperson at our February meeting - we are thrilled to have her at the helm of our board.

Karissa McKelvey is an open source programmer, who has been a core Dat team member for four years, and now a member of the Digital Democracy team. You may know her as @okdistribute online, and maybe you saw her Full Stack Fest Keynote. She managed the Dat desktop application and DatBase public registry projects. She is an innovative software developer as well as an accomplished writer, public speaker, and activist who works to support an equitable web. Formerly a research scientist at Indiana University, her work studying online political communication resulted in multiple peer-reviewed papers and press in outlets such as NPR and the Wall Street Journal. She has successfully led teams to success with diverse projects throughout her career in academia, non-profits, and industry. Her deep understanding of the technology and experience working on open web issues will bring valuable perspective to the board.

Kristen Ratan is co-Founder and Executive Director of the Collaborative Knowledge Foundation, a nonprofit with a mission to evolve how knowledge is created, produced, and shared. Kristen has a 20-year history working to accelerate advances in science and research communication, most recently as Publisher at the Public Library of Science (PLOS). Prior to that Kristen held leadership positions at HighWire Press, Atypon and BIOSIS. Kristen is on the Board of Directors of the American Institute of Physics Publishing and the nonprofit Community Resources for Science. She also serves on numerous advisory boards and industry committees.

Waldo Jaquith is with 18F’s State and Local Acquisitions practice, working with governments across the U.S. to help them better acquire software to serve the needs of the public. Previously, he was the Senior Advisor to the Sunlight Foundation, Director of the Knight-funded U.S. Open Data, ran the Knight-funded State Decoded project, and worked for the White House Office of Science and Technology. He serves as an advisor to and on the board of various organizations that sit at the intersection of technology and government.

Joshua M. Greenberg, PhD is joining as a board advisor. He is director of the Alfred P. Sloan Foundation's Digital Information Technology program. Dr. Greenberg received his Bachelor of Arts in History of Science, Medicine and Technology from the Johns Hopkins University, and both Masters and Doctoral degrees from Cornell University's Department of Science & Technology Studies. Before Sloan, he was Director of Digital Strategy and Scholarship at the New York Public Library, where he created the NYPL Labs team and launched a number of projects focused on deepening engagement through access to digital collections.

Join us in welcoming our new board members! We had a great time in San Francisco. We spent three straight days with our friends at the Internet Archive, got the chance to hang out with the Beaker Browser team, and linked up with many others.

Dat Protocol Working Group members at the Internet Archive: Tara, Joe, Bryan, Paul, and Danielle

Most of the Dat protocol working group at the @internetarchive! Missing @okdistribute and @mafintosh pic.twitter.com/JaAGqnlvK3

— Tara Vancil (@taravancil) February 1, 2018

Stay tuned for more updates. As always, you can reach out to Danielle and Joe at community@datproject.org anytime.

As part of the Dat in the Lab project we are working with different campuses in the University of California network. One of our goals is to publish researcher's data, code, and executable Linux container all as files in a version controlled Dat repository. For this to be useful, a person should be able to execute these Linux environments (aka containers) anywhere.

If you own your computer, or rent a cloud server, you can use sudo or root access to run containers. Most container systems like Docker expect you to have superuser privileges. In many university computing clusters (aka High-performance computing - "HPC"), hundreds of researchers share a single Linux environment and do not have the ability to install global software packages or execute commands that require sudo. Even workstations and laptops that are owned by an institution often do not allow users these privileges.

We are working with researchers who want to run a bioinformatics pipeline that was developed at UCLA on the cluster located at University of California, Merced. Both campuses have shared computing clusters running CentOS 7 Linux. However, the clusters are different. The UCLA cluster has thousands of software packages installed. These have been requested over time by UCLA researchers. These packages are not a standard part of the CentOS ecosystem and are not available to easily install at UC Merced.

The pipeline we are working with requires specific versions of Python, Bash, R and GCC to be installed and compiled. The UC Merced cluster does not have these versions available. The traditional way researchers have dealt with these code portability issues is by asking their cluster sysadmins to install all of the packages they need. Sometimes the sysadmins say no, as upgrading to (for example) Python 3 globally would break all Python 2 applications. Sometimes sysadmins say yes, but it takes a while as this is tedious work.

To address global upgrade breakages at UCLA they use the Environment Modules system to allow users to load in specific versions of packages into their environment. For example, when you first login to a compute node shell there will be no version of R loaded. But you can run a command like module load R/3.4.2 to get a specific version loaded into your path for that specific shell session. The UCLA pipeline we are working with has a series of module load directives executed at the beginning of the pipeline to ensure the correct environment is set up.

However, moving the pipeline to another cluster requires moving all of the modules as well. Some modules are single binary executables, easy to move to any other server with the same architecture, but others (such as R) are a complicated dependency graph including things like a certain version of an Intel Fortran compiler needed to compile R packages at runtime.

Moving the pipeline code is easy, but moving the underlying machinery (the specifically compiled and installed software dependencies and overall supporting Linux environment) can be very complicated. The worst part is, even if you seem to get the pipeline to work, there is no way to verify that you installed and compiled everything exactly the same on the new machine as it was installed on the old machine. You can only see if your program works, but there may be subtle bugs lurking that may go undetected due to an environmental difference (e.g. a slightly different C++ compiler being used which throws off the Fortran calculations in an obscure old version of an R package).

Our initial idea was to simply copy all of the modules from UCLA onto UC Merced, but this proved impossible. Running du -sh would have told us see how big the total installed size of all modules on the UCLA cluster was, but after many hours the command never finished. We needed another approach.

Containers

In theory, containers are a better way to address this problem. Instead of having everyone on a machine share a single Linux environment, containers allow for many environments to execute at the same time. The entire container can be version controlled for reproducibility, giving you cryptographically secure confidence that the code will run the same everywhere. Before we talk about containers, let's run over the basics of Linux.

Conceptually, Linux is made up of the kernel and user-space. The kernel interfaces with the physical hardware of the computer and presents an API (called system calls or syscalls) to interact with the hardware. Virtual machines (VMs) emulate everything from the hardware up, including the kernel, which provides a complete emulation of an environment but comes at the cost of performance, as your application is using e.g. virtual RAM which has to get mapped to physical RAM through multiple layers of abstraction. For a bioinformatics pipeline that might be designed to use 1TB of RAM, this will reduce performance and defeat the purpose of buying specialized hardware.

Containers differ from virtualization in that all containers running on one machine share the same kernel, which means they are not emulating any hardware or emulating the kernel. A program like VirtualBox or Xen does not qualify a container, but is rather classified as a VM because they emulate the whole machine. There are many ways to implement a container system depending on what your requirements are. A container system is any program that will use a set of syscalls to run a program in a Linux file system. What syscalls are used determine the security tradeoffs and kernel compatibility. You may have heard the phrase "everything in Linux is a file". This means that you can simply take a filesystem, throw it in a ZIP archive, and you have a container. How you execute the files as processes depends on which syscalls you use, but a container is essentially just a folder full of files and a way to run a process in that folder.

By renting a cloud machine, one can pick a distribution with the latest Linux kernel and use sudo without issue. Thereby giving access to every syscall, including the new shiny ones that modern container systems rely on. However researchers using shared university servers don't have control over the kernel and probably don't have sudo access.

To return to our goals: We want to publish researcher's data, code, and executable Linux container all as files in a version controlled Dat repository. We want anyone to be able to execute these Linux containers anywhere. To do this, it needs to be easy for researchers to set up and maintain containers on university machines without special sysadmin permissions.

Possible solutions

All of that stuff above leads to the real challenge of containers for reproducible science: making containers run everywhere, including restricted shared computing clusters. With that in mind let's look at different approaches to running containers in Linux.

chroot

The chroot syscall is sort of the proto-container. It's been around in Linux forever. In Linux, your filesytem starts at /. Usually a user puts their stuff somewhere like /home/alex. If you unzipped an Ubuntu distribution at /home/alex/ubuntu, you can use chroot to spawn a new process that is running relative to /home/alex/ubuntu but thinks it's at /. At it's most basic form, this lets you 'nest' filesystems and have different processes sharing the same kernel, but using different Linux environments. Using our definition above, this qualifies as a container. The downside is that there is no built-in security preventing things in the chroot from achieving root privileges, and you have to be root to create a chroot.

Root Daemon (Docker)

Docker runs a daemon process as root that, if you also have root, you can ask to download and boot containers for you. This means that with Docker, containers are executed as the root user. Also by default only other root users can talk to the Docker daemon at all. You can change the security so that users can run Docker without sudo, but this is not recommended by Docker as it means any user can execute root commands (e.g. Docker daemon access basically means root access).

The advantage of this approach is that it's simple. The Docker daemon manages the lifecycle of all containers running on a system. You use the Docker protocol to talk to the daemon, and the daemon translates that to whatever syscalls are available to it on the kernel it's running on. If you move to a different kernel, the Docker protocol stays the same (in theory, in practice this can break their protocol a lot which defeats the purpose of Docker in our opinion, but that's a different blog post). The downside is that the containers don't run as the user, they run as root. University sysadmins don't like this.

Rootless containers (runC, Charliecloud)

Since Linux Kernel 3.8 you have access to a syscall CLONE_NEWUSER which lets you create an user namespace as an unprivileged user. There's more info in this excellent blog post but the tl;dr is that you can, without sudo, create a process tree that maps your user on the actual system to a virtual user (like root) inside the new process namespace, meaning you can run processes as a "virtual" root without needing root yourself. Unfortunately in CentOS/RHEL user namespaces are still not enabled by default due to security concerns.

The Charliecloud project from Los Alamos has developed a minimal (~900 lines of C) chroot-style container system that is based on having user namespaces enabled.

CentOS appears to be is the only distribution that doesn't have user namespaces enabled. CentOS also happens to be popular in universities. In the long term, this seems like the best security for containers for shared environments. For now, to support CentOS as well as older servers/kernels, other solutions are still needed.

SetUID (Singularity, Bubblewrap)

If you don't have a way to do rootless containers, but you don't want to give your researchers root access, then you have to find a middle ground. Singularity is one such approach that works here. To install singularity, you need to be root, so that you can use the setuid syscall to 'mark' the singularity binary executable with root privileges. This means that singularity can do stuff as root, but the user's program cannot. As long as Singularity doesn't have any security exploits it's a nice tradeoff. Once you have Singularity installed, you can run containers without superuser privileges, and the container processes are owned by your user, not running as root.

Singularity will use user namespaces if they are available instead of setuid, so it works similarly to Docker in that it tries to provide a portal container execution context that works on a variety of new and old Linux distributions, but differs in that it doesn't require a central root daemon process, which makes it more popular with university sysadmins.

Bubblewrap is another similar approach that specifically tries to be a minimal setuid-based program that can spawn a container process and de-escalate all other privileges. The goal of bubblewrap is to keep surface area of the program minimal so that it can be audited more effectively, which is a great engineering principle that is (in my opinion) often overlooked. OpenSSL vs. Libsodium comes to mind.

Conclusion

It's worth mentioning the article that coined the phrase "containers don't contain" when discussing container security. In the university use case we described, we aren't looking for perfect security, but rather security that fulfills the requirements of university sysadmins.

So, which of the above approaches are we using at UC Merced? We've put in a request for the sysadmins there to install Singularity, which seems like the best approach since we can't run rootless containers on their CentOS. If they weren't running on CentOS we would probably just use rootless containers, as we could use them without asking the sysadmins to do anything. We'll do another blog post later in the year with more details on exactly what we're building with all this container stuff, but we wanted to take the opportunity here to talk about the scope of the challenge of deploying containers in the open science ecosystem.

Privacy on the web is an elusive thing. We often think our data is private only to later learn the company we trusted is using it for something we find unsavory (or it gets hacked and released en masse). In our view, this is one of the central shortcomings of the current system — centralized services are the only option for communities. This forces us to trust these services with our data, even if we do not want to or do not know how they are using it. For many, they view this as the cost of life online (which is now all but required for life offline). But it does not need to be.

In a recent post, we shared questions we ask before shipping software. Part of the post raised privacy concerns around hosting Wikipedia on Dat. Many people in our community asked: what does this mean for privacy on Dat in general? Should I be worried!? These are great questions! And thank you for keeping us accountable.

We apologize for raising these points and not addressing all of them (the last post was getting a bit long!). You can read more about reader privacy models specific to hosting large public datasets (like Wikipedia); this post will address how Dat privacy currently works and what we can do better.

Trust Someone

To live in this world we need to trust people (I will try to keep this grounded in reality and back to privacy soon!). When we get in a car, we trust other drivers to follow the rules. We trust that the planners and engineers did their job correctly. We trust the mechanic and car designers thought about our safety. If not, we trust our government to hold people accountable. We cannot engage in social life without trust.

The same is true for life online. There are two questions you have to answer when thinking about trust and privacy online:

• Who am I trusting?
• What information am I trusting them with?

For example, if you post on Facebook, you are trusting the company and employees of Facebook with your data. You are also trusting the people you have added as friends to not share your data. This feels strange because we have two very different groups to trust. On one hand, you have a community of friends where you have some measure of how much to trust them. On the other, you have to extend trust to Facebook without much to substantiate that trust (beyond our democratic institutions and social or economic pressures).

(There are systems that aim to minimize trust, such as Tor or blockchain systems. While these do allow for less trust, they also have other negative trade-offs. In order to decrease trust, you have to make the system more complex and slower. We seek to find a balance where users can decide on how much complexity they want for their security needs (you can use Dat over Tor!). For example, we want to trust the local public library without needing to connect via Tor or use a blockchain to checkout a book.)

To us, the question is not: how can we remove all trust from the system? Trust is human. The question is: how can we create communities where we can choose who to trust and with what information? Right now we are forced to opt-in to trusting organizations to connect online. We strive to make that optional, more explicit, and more flexible.

Foundation of Dat Privacy

The foundation of Dat privacy rests on private-by-default and end-to-end encryption for all metadata and content. Whenever you create and share information over Dat, only other users you explicitly share that information with can see it. There is no way other users can discover what you are sharing. This is as true for a "public" dat site and a private dat you are sharing with one person.

The second point, which we raised in the Wikipedia post, is how and when your IP address gets shared. This is where things get confusing because it depends on who you are connecting to and how you are connecting (and also where we can most improve documentation and implementation). We'll dive into different examples of how these work in practice.

Dat Privacy Examples

Private Peer-to-Peer

The simplest example, and default state for Dat, is private peer-to-peer sharing. In this case, we can imagine Joe wants to share some food pictures with Danielle. They discover each other using DNS and connect directly. No one else can see their IP addresses or Joes's food pictures. All metadata and content is private and encrypted in transit.

Key points in this example:

• Private Dat sharing has world class privacy (encrypted, no third-parties)
• Users sharing a common Dat trust other connections with your logs. Not all applications need a large pool of peers, as in our Wikipedia example.

Community Peer-to-Peer

Sharing data with a community of peers is a feature unique to Dat. It is a feature impossible to mimic on how the web works today. This is somewhere in between BitTorrent (global peer-to-peer) and one-to-one connections (client-server or direct peer-to-peer). As the network grows, we have to start being a bit more careful about the privacy implications.

In this example, I create a feed to share on Rotonde, a decentralized social network. When I share that key with my friends, they can re-share my feed. Similarly to how we trust our friends with Facebook data, we may be trusting our friends to keep the information private. (However, we can start to add more complexity when privacy needs to be guaranteed, with trade-offs).

These communities can be organized socially (sharing with friends I know IRL) or more organically as we've seen with Rotonde, where connections of connections start sharing feeds. Thus, trust in this situation becomes a people problem, in other words — messy.

Public Data

When considering publicly shared data, the chief concern is reader privacy (since we can assume that the data is okay to share widely). The two aspects of reader privacy are:

• Who can see what IP addresses are in the swarm?
• Can users in the swarm see specifically what content other users are requesting?

Concerns about sharing IP addresses may warrant its own post, as they become quite nuanced. IP addresses publicly identify your computer on a network. Whenever you connect to a website, they can view your IP address (and many analytics services store them). We want to minimize access to IP addresses and will continue to improve here. However, we need to be most careful in leaking IP addresses where there may be other surveillance or censorship concerns (such as in the Wikipedia example). Here again, we have to balance complexity and difficulty of use with common use cases — and maintain privacy in situations where it may be paramount.

The second question, of whether users can see what content is being requested, is similar to the difference between HTTPS and HTTP. Without HTTPS sites, ISPs (or other middlemen) can see exactly what page of wikipedia.org you request, whereas with HTTPS they only see a connection to wikipedia.org. Our reader privacy on the p2p web post covers all the nuances, but we'll address some key points here.

Dat with "Old Web" Security

The first point is that Dat can always be at least as secure as the "old web." That is, trusting one centralized organization with IP logs and information about what users are requesting.

Dat can be used over HTTP to act as a traditional web service (with the added benefit of having versioned websites and verifying content isn't corrupted between the source and the user). We can also have applications that connect to trusted peers directly, mimicking a client-server model but over the peer-to-peer network.

Peer-to-Peer

Public data with peer-to-peer distribution will be most useful in cases where we need to distribute bandwidth. As before, we can continue to increase complexity to add more privacy (see reader privacy post for more):

• Get data from a trusted sets of peers, i.e. Web of Trust.
• Make it more difficult for single users to "win" discovery, ensuring a more unique distribution of peers in each swarm.
• Mask content requests by adding in noise, making it harder to identify requested content.

Protocol Options

Beyond different peer-to-peer models, we can start using different protocols. Dat gives users control over how to discover & connect to peers, giving more flexibility compared to the existing web. For example, you can run Dat in an offline network where all connections stay in the local network. This flexibility is what allows us to run Dat over HTTP connections.

Being protocol agnostic allows for more flexibility in security models. But with this customization we also have make sure security and privacy implications of different implementations are clear, an area we can really improve on.

Where We Can Improve

Thinking through this post and the questions we got, we see a few areas we can definitely improve:

• Improve our DHT module so IP addresses are more secure
• Provide option to opt-out of peer-to-peer for public data applications (i.e. use the web's traditional trust model)
• Demonstrate how Dat over HTTP or other "old web" and trusted peer models can work via Dat.
• Improve transparency! Make nice pictures that show how things work; this stuff is complicated.

Most importantly, we can talk about privacy and security more! We can ask hard questions. This is an area we feel the old web really falls short and we are still building models for how this can work in the future. The balance will always be between complexity, speed, usability, and privacy. We want to be able to shift this balance as necessary for different applications but always make it clear what the trade-offs are.

Thanks to the community for bringing up the privacy questions. Let us know if you have more on Twitter or our chatroom.