Big data is a well known problem and it is currently being stored it on clouds and
processed on super computers with burdensome resources (RAM memory, storage space,
and processing power).
We invented and fully implemented new Web technology that allows us to process big
data on a regular laptop, tablet, or phone inside a Browser based on the user’s needs –
not the application’s limitations. This technology has no analogs that currently exist in the world today.
To achieve this power of web applications, we found a way to process millions of
records extremely fast and without any server support. As a result, our technology
accelerates the Web application, reduces network bandwidth requirements, and distributes
the server load to thousands and millions of users' computers.
No new hardware, programming languages, plugins, nor are any other resources required.
- User is defined as the individual who uses the web application.
- Client is part of the application program and located in the Browser on the user’s computer.
The Client is getting the data for the user from the Server, a computer that stores data and processes this data.
- Bandwidth is the speed of data delivery in the network between the Client and Server.
- Offline data processing is the ability of the client to work inside the
Browser without Server support and even without a connection to the Internet (network).
- Big data (or Megadata) is a term used to describe an overwhelming amount of data
(tables with millions of records) - the most popular phrase in the US for the past few years.
- Responsive Design is web application implementation that can run on both mobile devices and stationary web devices (regular computers)
What Everybody Does
Currently, many popular program libraries were created for simplifying and accelerating
the development process of applications. These libraries were not geared towards processing
big data efficiently. This is why most web applications become slow, complicated,
and require permanent server support.
Now every web user has a powerful, multi-processor computer but big data web business
applications do not use these free and endless resources because popular web
technologies have four major restrictions:
- Slow delivery of large amounts of data (Extremely critical for mobile devices with lower Bandwidth).
- Client side performance limitations (Browser can process less data and is much slower than the Server).
- User computer memory restrictions.
- The Client’s inability to process data inside the Browser offline.
What We Do
We found a way to resolve all these problems by developing the following improvements:
- Developed compact data representation with pack and unpack algorithms for faster data delivery.
- Quantum leap performance of data processing (We can sort 1,000,000 records in one second).
- Developed an advanced compact data format to avoid memory restrictions even for mobile devices.
- Found a way for endless data processing offline.
In short, we created an advanced data processing engine with an emphasis on simplicity, usability,
performance, and maximum functionality without the problems inherent with other solutions.
Our technology provides a business application with all the necessary features
and intuitive controls for processing big data. It is customizable, straightforward,
transparent, light, and intuitive. For “desktop” web users, we benefit them with
even more processing power for having a computer with a bigger monitor, more RAM,
and a faster processor.
Most business applications process big data on servers. We can shift this to
the client instead of delegating it to an overloaded server. Thousands of
user computers can simultaneously simplify and accelerate the whole application
and hence, not rely on the number of server processors available to the users.
Our technology is compatible with all standard browsers (Chrome, Firefox, Edge, Safari,
etc.) and can be integrated easily and seamlessly into any existing web application.
It is fully implemented, debugged, and being installed on a variety of Internet
Who Does This Benefit?
Corporate IT/MIS/DBA support groups, web application developers (in-house and OEM),
and knowledge workers - analysts, engineers, scientists, planners, managers,
salespeople, researchers, teachers, students, etc. – for whom the ability
to access data quickly, directly relates to their productivity for marketing
optimization, portfolio and risk analysis, software analytics, operational
intelligence, fraud prevention, etc…
Megadata Web is a powerful addition for all web-based big data analytics
platforms and for all types of organizations, from startups to Fortune 500 companies.
Why Is This Important?
According to the analyst firm Gartner, distributed off-line client processing is
the future of the Internet.
By transferring the data processing overhead from the server to the client,
the user’s ability to view and freely process information is significantly
enhanced while simultaneously increasing Internet access and server throughput.
How Fast Is Fast?
By eliminating navigation across multiple screens, Megadata Web significantly
boosts the responsiveness of the application being used.
The average execution time for Megadata Web sorting in different browsers
(not IE*) for 1,000,000 records is 1 second!
Technology has dramatically increased, by up to 10,000 times, the amount of
data/information that can be delivered onto just a single Web page.
Through the use of our unique data-compacting methods, users will
never again see the ubiquitous, unproductive query response:
"first 20 of 4,732 records" on their Web browsers.
Megadata Web reduces, by up to 98%, the redundant and unnecessary
network traffic associated with current database querying methods.
This translates into increased network capacity for future expansion,
and lower maintenance and MSP expenses.
Is It Easy To Implement?
- Compatible and interoperable with all standard browsers (Chrome, Firefox,
Edge, Safari, Opera)
- Integrates seamlessly with 3rd party server databases, middleware platforms,
Operating Systems and technologies (e.g., .NET, PHP, Java, Node.js…).
- Requires no additional client-side software or plug-ins
- No access to local hard drive needed
When Will It Be Available?
Megadata Web is fully implemented, debugged, and being installed on a
variety of Internet client/server applications.
Some companies that use Megadata Web technology include:
- Guardian Analytics, Inc. (online banking fraud prevention)
- Novartis Institute Functional Genomics (science for preclinical drug discovery)
- VMWare, Inc. /Cetas Software, Inc. (big data analytics)
- Secure Campus, Inc. (network security)
Alisa, launched in June 2016, is a web service provided by Megadata Web for data management.
It can be used for visualization of big (up to 16,000,000
records) data tables in one chunk that
Browser users can load from a local hard drive and it's a secure process on the computer off-line.
No data upload is required! Compare this to Microsoft Excel which processes a maximum of
records in one table.
The web service provides a means for visualizing data with pie charts, bar charts, lineplots,
scatterplots, line charts, donuts, and timelines. Data is imported and exported in a comma-separated
values file format.
Snapshots (click to zoom):
Margo, launched in September 2016, is a web service provided by Megadata Web
for DNA Codon Usage Calculation in nucleotide sequences (FASTA format files).
It can be used for codon usage calculation in big
(up to 4 GB
) FASTA files
that Browser users can load from a local hard drive and it's a secure process
on the computer off-line. No data upload is required!
Snapshots (click to zoom):
Alyona, launched in July 2016, is a web service provided by Megadata Web for viewing,
searching, analyzing, and data mining VCF (Variant Call Format) files. It can be used
for processing of super big (up to 100 GB
) VCF files in multiple chunks with
up to 1,000,000 records per chunk
that Browser users can load from a local hard
drive and it's a secure process on the computer off-line. No data upload is required!
Because every chunk of data is processed separately from other chunks, Alyona
can process super big data sets relatively fast.
Snapshot (click to zoom):
Anjela, launched in August 2016, is a web service provided by Megadata Web for searching
motifs in nucleotide sequences (FASTA format files). It can be used for searching motifs
FASTA files that Browser users can load from a local hard drive and it's a secure
process on the computer off-line. No data upload is required!
Motif Finder Anjela (Serching 20 motifs - some of them represent up to 2^45 patterns - in chromosome #1 (234.3 MB).)
Snapshot (click to zoom):
Anastasia, launched in October 2016, is a web service provided by Megadata
Web for data management. It can be used for visualization of super big
(up to 200,000,000
records) data tables in multiple chunks with
up to 1,000,000 records per chunk that Browser users can load from a
local hard drive and it's a secure process on the computer off-line.
No data upload is required! Compare this to Microsoft Excel which
processes a maximum of only 1,048,576
records of one table.
Because every chunk of data is processed separately from other chunks,
Anastasia can process super big data sets relatively fast.
Data is imported and exported in a comma-separated values file format.