FAQ
?I am not sure how the 2GB file might expand when you read it into R. I
would suggest that you take a portion, e.g., 500MB, and read it in and see
how large the resulting object is in R. Continue and process this smaller
size to see how memory utilization changes. This will provide information
as to how much memory you might need.


If you fail reading in a large file, start cutting it half to see what you
can get in and process. Do you need all the columns that might be in the
data, or can you remove some? Can you sample the data and perform the
calculation on that subset to get a rational answer. Can you put the data
in a database (SQLite) and then pull in selected records more easily.
There are several other alternatives you might want to consider, but at
least find out what you can process on you available system and then you
might know what some of the options are. Get access to a server with
32-64-128GB to see if you can process the data the way you want if you had
a larger system.




Jim Holtman
Data Munger Guru


What is the problem that you are trying to solve?
Tell me what you want to do, not how you want to do it.


On Mon, Nov 23, 2015 at 2:26 PM, Tamsila Parveen wrote:

I already tried on windows to increase virtual memory, but it didn't work.
Actually I want to analyze MD-Simulation trajectories through R and trying
to generate cross correlation, PCA plotting graphs, but R didn't accept
file of 2GB. Even though my system is of 4GB RAM.



On Monday, 23 November 2015, 4:53, jim holtman wrote:


My general rule of thumb is that I should have 3-4 times as much RAM as
the largest object that I am working with. So hopefully you have at least
4 GB of RAM on your system. Also exactly what processing (packages,
functions, algorithms, etc.) are you using. So functions may create
multiple copies, or they may create temporary objects bigger than the
original. So help us out and provide more information. You might be able
to add virtual memory, but this may slow down your process quite a bit with
paging. If you do go this direction, then learn how to use the performance
monitoring tools on your system to see what is happening.?


Jim Holtman
Data Munger Guru

What is the problem that you are trying to solve?
Tell me what you want to do, not how you want to do it.

On Sun, Nov 22, 2015 at 10:08 AM, Tamsila Parveen via R-help <
r-help at r-project.org> wrote:

Hello, Is there anyone to help me out how can I resolve memory
issue of R, when I want to analyze data of 1Gb file, R returns me Error:
not allocate of vector size of 1.8 GB.I tried on linux as well as on
windows with 64 bit system and using 64 bit R-3.2.2 version. So anyone who
knows please guide me to resolve this issue
[[alternative HTML version deleted]]

______________________________________________
R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
<http://www.r-project.org/posting-guide.html>
and provide commented, minimal, self-contained, reproducible code.




  [[alternative HTML version deleted]]

Search Discussions

Discussion Posts

Previous

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 5 of 5 | next ›
Discussion Overview
groupr-help @
categoriesr
postedNov 22, '15 at 3:08p
activeNov 24, '15 at 2:07a
posts5
users4
websiter-project.org
irc#r

People

Translate

site design / logo © 2018 Grokbase