Trouble Inputting a lot of Data into Minitab

[TooMuchData]

Hi,

My trouble right now is that I have 78 Million observations to input into Minitab. Luckily they all follow a pattern. However, every site I've searched for help with my issue has suggested the autofill (drag and drop) for this issue. However, even the autofill for 78 Million observations would be much too time consuming and painful.

Is there any way I can specify for say columns 1-10,000,000 = 0. And after that if I need for the next 20 million columns to start at the value 1 and increase by say .5 every column until the 20 millionth column, is there a way to do that?

I've been struggling with this for awhile now, and can't seem to figure it out. Any help would be greatly appreciated.

Thank you,

TooMuchData

P.S. I'm running Minitab 16.

[Miner]

If I understand correctly, you should be able to do this using Calc > Make Patterned Data > Simple Set of Numbers.

[TooMuchData]

Hi Miner,

I tried that, but it said "*Error* Too many data items". Is minitab just not able to analyze a data series this large?

Thanks for the help,

TooMuchData

[Berg.Jlle]

In Reply to Parent Post by TooMuchData

Hi,

My trouble right now is that I have 78 Million observations to input into Minitab. Luckily they all follow a pattern. However, every site I've searched for help with my issue has suggested the autofill (drag and drop) for this issue. However, even the autofill for 78 Million observations would be much too time consuming and painful.

Is there any way I can specify for say columns 1-10,000,000 = 0. And after that if I need for the next 20 million columns to start at the value 1 and increase by say .5 every column until the 20 millionth column, is there a way to do that?

I've been struggling with this for awhile now, and can't seem to figure it out. Any help would be greatly appreciated.

Thank you,

TooMuchData

P.S. I'm running Minitab 16.

Hi.

As far as I know, each worksheet in Minitab 16 may contain up to 4,000 columns and each column may contain up to 10,000,000 values.

regards.

[Barbara B]

A Minitab worksheet can contain more than 10,000,000 rows, but "only" the first 10,000,000 are shown.

To get a column with 10,000,000 times "0" first and afterwards patterned data starting from 0.5 to 5,000,000 with an increment of 0.5 you can stack some columns which contain up to 10,000,000 values. A more convenient way than the menu (Calc > Make Patterned Data > Simple Set of Numbers like Miner mentioned) is to use the Command Line Editor (opens with CTRL + L, see attached screenshot), paste the following code and click on "Submit"

Set c1
 1( 0 : 0 / 1 )10000000
 End.
Set c2
  1( 0.5 : 4999999 / 0.5 )1
  End.
Set c3
  1( 4999999.5 : 5000000 / 0.5 )1
  End.

Column "c1" contains 10,000,000 times "0", column "c2" patterned data from 0.5 to 4,999,999 (9,999,998 values total) and column "c3" has two values (4,999,999.5 and 5,000,000). The "make patterned data" command will only work properly up to 9,999,999 values if an increment is set (like 0.5).

No we have 20,000,000 values in the worksheet in three different columns and want to stack them into one columne (here: column "c4"). This could be done via
Data > Stack > Columns
or with the following code lines in the Command Line Editor:

Stack  C1-C3 c4.

The worksheet doesn't show all the 20,000,000 values, but they are stored. Just click on the "i" button (see attached screenshot) or count the non-missing values in column c4
Stat > Basic Statistics > Show Descriptive Statistics
or with the Command Line Editor

Describe C4;
  N.

I'm not sure if Minitab will work with 78,000,000 values properly. Maybe you can post your experience with your huge data set?

Regards,

Barbara

[Allattar]

Well past 10million rows gets a little odd.

The worksheet appears to carry on below 10million, but you cant scroll below that row.