The core of the core of the big data solutions -- Map

Title:       The core of the core of the big data solutions -- Map
Author:      pengwenwei
Email:       pww71@foxmail.com
Language:    c++
Platform:    Windows, linux
Technology:  Perfect hash algorithm
Level:       Advanced
Description: Map algorithm with high performance
Section      MFC c++ map stl
SubSection   c++ algorithm
License:     (GPLv3)

    Download demo project - 1070 Kb
    Download source - 1070 Kb

Introduction:
For the c++ program, map is used everywhere.And bottleneck of program performance is often the performance of map.Especially in the case of large data,and the business association closely and unable to realize the data distribution and parallel processing condition.So the performance of map becomes the key technology.

In the work experience with telecommunications industry and the information security industry, I was dealing with the big bottom data,especially the most complex information security industry data,all can’t do without map.

For example, IP table, MAC table, telephone number list, domain name resolution table, ID number table query, the Trojan horse virus characteristic code of cloud killing etc..

The map of STL library using binary chop, its has the worst performance.Google Hash map has the optimal performance and memory at present, but it has repeated collision probability.Now the big data rarely use a collision probability map,especially relating to fees, can’t be wrong.

Now I put my algorithms out here,there are three kinds of map,after the build is Hash map.We can test the comparison,my algorithm has the zero probability of collision,but its performance is also better than the hash algorithm, even its ordinary performance has no much difference with Google.

My algorithm is perfect hash algorithm,its key index and the principle of compression algorithm is out of the ordinary,the most important is a completely different structure,so the key index compression  is fundamentally different.The most direct benefit for program is that for the original map need ten servers for solutions but now I only need one server.
Declare: the code can not be used for commercial purposes, if for commercial applications,you can contact me with QQ 75293192.
Download:
https://sourceforge.net/projects/pwwhashmap/files

Applications:
First,modern warfare can’t be without the mass of information query, if the query of enemy target information slows down a second, it could lead to the delaying fighter, leading to failure of the entire war. Information retrieval is inseparable from the map, if military products use pwwhashMap instead of the traditional map,you must be the winner.

Scond,the performance of the router determines the surfing speed, just replace open source router code map for pwwHashMap, its speed can increase ten times.
There are many tables to query and set in the router DHCP ptotocol,such as IP,Mac ,and all these are completed by map.But until now,all map are  using STL liabrary,its performance is very low,and using the Hash map has error probability,so it can only use multi router packet dispersion treatment.If using pwwHashMap, you can save at least ten sets of equipment.

Third,Hadoop is recognized as the big data solutions at present,and its most fundamental thing is super heavy use of the map,instead of SQL and table.Hadoop assumes the huge amounts of data so that the data is completely unable to move, people must carry on the data analysis in the local.But as long as the open source Hadoop code of the map changes into pwwHashMap, the performance will increase hundredfold without any problems.


Background to this article that may be useful such as an introduction to the basic ideas presented:
http://blog.csdn.net/chixinmuzi/article/details/1727195
0
pww71
3/30/2015 9:46:22 AM
embarcadero.algorithms 211 articles. 0 followers. Follow

0 Replies
681 Views

Similar Articles

[PageSpeed] 9

Reply:

Similar Artilces:

mozilla/testing/performance/talos/page_load_test/jss core-eval-1.html core-eval-10.html core-eval-11.html core-eval-12.html core-eval-13.html core-eval-14.html core-eval-15.html core-eval-16.html core
Diffs: http://bonsai.mozilla.org/cvsview2.cgi?command=DIRECTORY&subdir=mozilla/testing/performance/talos/page_load_test/jss&files=core-eval-1.html:core-eval-10.html:core-eval-11.html:core-eval-12.html:core-eval-13.html:core-eval-14.html:core-eval-15.html:core-eval-16.html:core-eval-2.html:core-eval-3.html:core-eval-4.html:core-eval-5.html:core-eval-6.html:core-eval-7.html:core-eval-8.html:core-eval-9.html:jss.manifest:object-array-1.html:object-array-10.html:object-array-11.html:object-array-12.html:object-array-13.html:object-array-14.html:object-array-15.html:object-array-16.html:obj...

Core Temperatures In 4-Core Processor
I figure that someone here will have an idea about my conundrum. I built a fairly plain-vanilla computer for HTPC use. The processor is an Intel Q6600. The system runs pretty much at 100% on all cores running a BOINC distributed-processing application 24X7 so, given the low-profile HTPC case the temperatures inside are of some concern. I run the program Core Temp to keep an eye on the processor and have found that the cores run at wildly different temperatures. For example just a minute ago I find that one core is at 67C and another is a 56C with the other two in the middle. H...

Replace Netware core with Linux Core
Hi, I am investigating the things needed to replace servers with Netware core (Netware 6.5) with Linux Core (SLES9). The servers run various services, some of which do not have a Linux version afaik. I created the following table: Netware server --> Linux equivalent iManager --> Linux version available Novell eDirectory --> Linux version available Novell GroupWise 6.5 --> Upgrade to GroupWise 7 Norman Firebreak --> Norman Virus Control on Linux HP Proliant tools (e.g. Insight Manager agents) --> SLES9 tested and approved version available Problems AFAI...

quad core or two dual core xeon
I have read some consider two quad core processors to be overkill. Processor clock speed being equal, would you think it better to have an oes2 server with a single quad core processor or two dual core processors? Thanks, Ed B. Hi, Ed Brennan wrote: > > I have read some consider two quad core processors to be overkill. That's a nice understatement. Even a single core of todays CPUs is overkill for most servers. > Processor clock speed being equal, would you think it better to have an > oes2 server with a single quad core processor or two dual core p...

only detecting 1 core on quad core laptop
I just installed Suse 11.1 (64 bit) on my quad core (Core 2 Quad Q9000) laptop. When I cat /proc/cpuinfo it only reports 1 core and a lower cpu frequency than I expected. I do not have acpi=off in my boot parameters (as suggested as an issue in other posts I have come across). Below is cpuinfo: processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 23 model name : Intel(R) Core(TM)2 Quad CPU Q9000 @ 2.00GHz stepping : 10 cpu MHz : 1600.000 cache size : 3072 KB physical id : 0 siblings : 1 core id ...

Extending and altering core functionality without modifying the core?
I am creating a virtual host system with DNN.  I am creating a templated site with all the technologies of my web app in the pages and modules I am creating as a portal master template.  My users will be issued a DNN portal derived from my feature rich template, and begin running their copy of my web app. I am looking for the ability to lock down certain features of DNN from the admin. I want my admin user (owner of portal, but not host) to have a limited set of DNN features. For example, no access to Authentication settings.  I would want to add my own new items, such as mor...

core tests using CPAN / non-core facilities
folks, wrt my recent B::Concise patches, I was pondering trying to convert ext/B/t/OptreeCheck.pm to something on CPAN, perhaps Devel::Optree::RegressionTest. Leaving aside the questions of what to name it, and what to put in it, etc.. Id like to focus on these Qs Q1: whether elective tests are wise / allowed. It would be bad if tests were routinely skipped in devel builds. # optree_samples.t use Test::More # this isnt allowed !? eval { use Devel::Optree::RegressionTest }; if ($@) { carp "install the damn thing" if $pedantic and $Config{usedevel}; ...

Core Code Question (core team, please look)
On line 553 of ModuleSettings.ascx.vb ctlSpecific.ID = src.Substring(src.LastIndexOf("/")) Is there a reason behind naming the control in this manner? I am trying to implement a third party ASP.NET control (ComponentArt) that uses the control Naming path as a basis for its script function names.  the / and the . caused by the above line (your control ends up having an Id similar to "/mysettingscontrol.ascx" ) is causing all kinds of JS fits.  (you can't have / or . in a function name) I am going to try to get around this by using this.ID=this.ID.Replace("/","").Replace(".","_"...

Concerning smoke-core/core/update-miscellaneous-modules
This commit removes a test added in blead by commit d1718a7cf5: From 71e2de81b7964c7258ec1ebec3aba09a9a7533a4 Mon Sep 17 00:00:00 2001 From: Chris 'BinGOs' Williams <chris@bingosnet.co.uk> Date: Fri, 8 Jun 2012 13:20:11 +0100 Subject: [PATCH] Updated Safe to CPAN release 2.33 --IJpNTDwzlM2Ie8A6 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jun 08, 2012 at 03:29:10PM -0000, Father Chrysostomos wrote: > This commit removes a test added in blead by commit d1718a7cf5: >=20 > >F...

Core Team: An idea for updating DNN core and modules
Ok, maybe this is a pipe dream, but I find that I spend a lot of time trying to keep track of updates to any modules I have installed. The DNN core isn't such a problem, but here's a thought: Is there a way to incorporate some functionality into the core that will allow you to configure it to look for updates much like Windows notifies you if there are critical updates. Then whenever there are core changes, the host or admin receives a message saying that updates are available. Even better, if this could be integrated into a module. So, as part of the module code the creator specifi...

Idea for modifying DNN Core behavior without Core Changes
I am currently working on a portal that describes DNN 2.0 in time slices. For example, 1) what happens when the DNN app is started 2) follow the path of a request through the DNN app 3) injection of modules etc. Once I get this a little more fleshed out, I'll post a link here for those interested. While looking at Global.asax I noticed the timers that are used for UsersOnline and the daily log purging. This made me wonder: has the core team considered a collection of application level objects (UserOnline timer, Daily timer) instantiated from a list in the web.config at ...

Retreiving data from CORE tables
hi all, I would like to use some fields of the user table of DNN(core) and put it in a custom tab named MyAccount. I'm wondering since all stored procedures related to this table are already made wih the core if it is possible to use those datas without creating a custom module with dataprovider, sqldataprovider, bll and all the stuff that is necessary to a custom module. Could I for example only use a .ascx and retreive the data with the provider made by the core from the user table (in the page load). And I dont want to change the core either. My goal would be after a user is logged...

Precedence bug ? Why does my CORE::GLOBAL::ref have the same precedence as CORE::ref ?
After some discussion at comp.lang.perl.misc: There seems to be a precedence problem when overriding builtin's: <perl> #!perl -w package CORE::GLOBAL; use subs qw(ref); sub ref(;$) { my $type = CORE::ref( shift || $_ ); $type =~ s/^(?:main)?:://; return $type; } package main; my $dummy = 'Drives'; @_ = ( It, $dummy , Me, Nuts ); use Data::Dumper; my $self = CORE::ref $_[0] ? [ FIRST, 1 ] : { SECOND => 2 }; my $self1 = ref $_[0] ? [ FIRST, 1 ] : { SECOND => 2 }; print Dumper $self, $self1; </perl> > p...

Ensuring TIdServer threads are spawned one a particular core or group of cores
How can this be done thanks :) Colin wrote: > How can this be done Assuming you are referring to Indy's TIdTCPServer component, then you can derive a new class from TIdThreadWithTask and override its virtual AfterConstruction() or BeforeRun() method to assign a thread instance to the desired core/group using the Win32 API SetThreadAffinityMask() and SetThreadGroupAffinity() functions as needed. Then you can assign one of the TIdSchedulerOfThread... components to the TIdTCPServer::Scheduler property and assign your custom thread class to the TIdSchedulerOfThread:...

Any way to get application to utilize more than one core in multi-core setup?
Hi, I have a command line OCR program called OCR Shop XTR (Vividata corp) that I am using on a system with a 6-core AMD chip. I changed the bios so that the 6-cores were activated, but htop shows me that while the program is running, I am only getting activity on one core (the program maxes out the one core with consistent usage between 97% and 100%). I have read that many programs are not written to take advantage of multiple core cpu's. However, I am just hoping that there is some way to get this program to take advantage of the extra cores. Does anyone know of a way to i...

Web resources about - The core of the core of the big data solutions -- Map - embarcadero.algorithms

Solution - Wikipedia, the free encyclopedia
In chemistry , a solution is a homogeneous mixture composed of only one phase . In such a mixture, a solute is a substance dissolved in another ...

'Hungry' Jack Cowin's solution to penalty rate stand-off
Hungry Jack's founder Jack Cowin says consumers will have to pay more for fast food if campaign to reduce penalty rates fails.

Why Canada's refugee plan falls well short of a real solution
Simply picking the cream of the crop from the world's swelling refugee population and bringing them to rich countries like Canada fails to deal ...

Audiodraft Offers A Custom-Made Solution For Advertising And Music Industry
Music represents a major part in the success of a commercial clip or a movie, and AudioDraft from Helsinki offers a tailored solution – no matter ...

Tuesday Evening Open Thread: Spoiler Solution
... and yet you’re considerate enough to avoid dropping spoilers on those of us who haven’t seen it yet. We appreciate it! But here’s a solution ...

New breach detection solution alerts enterprises to insider threats
... at Eastwind points out, "Protecting the perimeter remains critical to a defense-in-depth strategy. However, we read daily how these solutions ...

Two must-have MacBook power solutions for up to 76% off
Nothing good last forever, and that includes your MacBook Air power adapter. That’s why we make sure to grab an extra whenever it’s on sale—like ...

Frank Bruni thinks diversity is a fraud... but his solution is questionable.
... schools should "also pay greater heed to how gagged so many politically conservative faculty members and students feel." That's not a solution, ...

Afghanistan: Another Broken Obama Promise; Taliban Poised to Retake Sangin; Money Trail Shows Billions ...
... offer only criticism without an alternative plan. I happen to have the very face-saving plan the president needs. Mish 5-Point Proposed Solution ...

Donald Trump gloated about the Miss Universe mix-up, and offered an unusual solution
... (@realDonaldTrump) December 21, 2015 Trump's thoughts didn't end there. This morning he went on the "Today" show and offered an unusual solution ...

Resources last updated: 12/24/2015 8:20:48 AM